{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 05 - Data Preparation and Advanced Model Evaluation\n",
    "\n",
    "by [Alejandro Correa Bahnsen](http://www.albahnsen.com/) & [Iván Torroledo](http://www.ivantorroledo.com/)\n",
    "\n",
    "version 1.2, Feb 2018\n",
    "\n",
    "## Part of the class [Machine Learning for Risk Management](https://github.com/albahnsen/ML_RiskManagement)\n",
    "\n",
    "\n",
    "This notebook is licensed under a [Creative Commons Attribution-ShareAlike 3.0 Unported License](http://creativecommons.org/licenses/by-sa/3.0/deed.en_US). Special thanks goes to [Kevin Markham](https://github.com/justmarkham)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Handling missing values\n",
    "\n",
    "scikit-learn models expect that all values are **numeric** and **hold meaning**. Thus, missing values are not allowed by scikit-learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PassengerId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Survived  Pclass  \\\n",
       "PassengerId                     \n",
       "1                   0       3   \n",
       "2                   1       1   \n",
       "3                   1       3   \n",
       "4                   1       1   \n",
       "5                   0       3   \n",
       "\n",
       "                                                          Name     Sex   Age  \\\n",
       "PassengerId                                                                    \n",
       "1                                      Braund, Mr. Owen Harris    male  22.0   \n",
       "2            Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0   \n",
       "3                                       Heikkinen, Miss. Laina  female  26.0   \n",
       "4                 Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0   \n",
       "5                                     Allen, Mr. William Henry    male  35.0   \n",
       "\n",
       "             SibSp  Parch            Ticket     Fare Cabin Embarked  \n",
       "PassengerId                                                          \n",
       "1                1      0         A/5 21171   7.2500   NaN        S  \n",
       "2                1      0          PC 17599  71.2833   C85        C  \n",
       "3                0      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "4                1      0            113803  53.1000  C123        S  \n",
       "5                0      0            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import zipfile\n",
    "with zipfile.ZipFile('../datasets/titanic.csv.zip', 'r') as z:\n",
    "    f = z.open('titanic.csv')\n",
    "    titanic = pd.read_csv(f, sep=',', index_col=0)\n",
    "titanic.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Survived      0\n",
       "Pclass        0\n",
       "Name          0\n",
       "Sex           0\n",
       "Age         177\n",
       "SibSp         0\n",
       "Parch         0\n",
       "Ticket        0\n",
       "Fare          0\n",
       "Cabin       687\n",
       "Embarked      2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check for missing values\n",
    "titanic.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One possible strategy is to **drop missing values**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(183, 11)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# drop rows with any missing values\n",
    "titanic.dropna().shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(714, 11)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# drop rows where Age is missing\n",
    "titanic[titanic.Age.notnull()].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes a better strategy is to **impute missing values**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "29.69911764705882"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# mean Age\n",
    "titanic.Age.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# median Age\n",
    "titanic.Age.median()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    24.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# most frequent Age\n",
    "titanic.Age.mode()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# fill missing values for Age with the median age\n",
    "titanic.Age.fillna(titanic.Age.median(), inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another strategy would be to build a **KNN model** just to impute missing values. How would we do that?\n",
    "\n",
    "If values are missing from a categorical feature, we could treat the missing values as **another category**. Why might that make sense?\n",
    "\n",
    "How do we **choose** between all of these strategies?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Handling categorical features\n",
    "\n",
    "How do we include a categorical feature in our model?\n",
    "\n",
    "- **Ordered categories:** transform them to sensible numeric values (example: small=1, medium=2, large=3)\n",
    "- **Unordered categories:** use dummy encoding (0/1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PassengerId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Moran, Mr. James</td>\n",
       "      <td>male</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330877</td>\n",
       "      <td>8.4583</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>McCarthy, Mr. Timothy J</td>\n",
       "      <td>male</td>\n",
       "      <td>54.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>17463</td>\n",
       "      <td>51.8625</td>\n",
       "      <td>E46</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Palsson, Master. Gosta Leonard</td>\n",
       "      <td>male</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>349909</td>\n",
       "      <td>21.0750</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)</td>\n",
       "      <td>female</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>347742</td>\n",
       "      <td>11.1333</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>Nasser, Mrs. Nicholas (Adele Achem)</td>\n",
       "      <td>female</td>\n",
       "      <td>14.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>237736</td>\n",
       "      <td>30.0708</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Survived  Pclass  \\\n",
       "PassengerId                     \n",
       "1                   0       3   \n",
       "2                   1       1   \n",
       "3                   1       3   \n",
       "4                   1       1   \n",
       "5                   0       3   \n",
       "6                   0       3   \n",
       "7                   0       1   \n",
       "8                   0       3   \n",
       "9                   1       3   \n",
       "10                  1       2   \n",
       "\n",
       "                                                          Name     Sex   Age  \\\n",
       "PassengerId                                                                    \n",
       "1                                      Braund, Mr. Owen Harris    male  22.0   \n",
       "2            Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0   \n",
       "3                                       Heikkinen, Miss. Laina  female  26.0   \n",
       "4                 Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0   \n",
       "5                                     Allen, Mr. William Henry    male  35.0   \n",
       "6                                             Moran, Mr. James    male  28.0   \n",
       "7                                      McCarthy, Mr. Timothy J    male  54.0   \n",
       "8                               Palsson, Master. Gosta Leonard    male   2.0   \n",
       "9            Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)  female  27.0   \n",
       "10                         Nasser, Mrs. Nicholas (Adele Achem)  female  14.0   \n",
       "\n",
       "             SibSp  Parch            Ticket     Fare Cabin Embarked  \n",
       "PassengerId                                                          \n",
       "1                1      0         A/5 21171   7.2500   NaN        S  \n",
       "2                1      0          PC 17599  71.2833   C85        C  \n",
       "3                0      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "4                1      0            113803  53.1000  C123        S  \n",
       "5                0      0            373450   8.0500   NaN        S  \n",
       "6                0      0            330877   8.4583   NaN        Q  \n",
       "7                0      0             17463  51.8625   E46        S  \n",
       "8                3      1            349909  21.0750   NaN        S  \n",
       "9                0      2            347742  11.1333   NaN        S  \n",
       "10               1      0            237736  30.0708   NaN        C  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# encode Sex_Female feature\n",
    "titanic['Sex_Female'] = titanic.Sex.map({'male':0, 'female':1})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create a DataFrame of dummy variables for Embarked\n",
    "embarked_dummies = pd.get_dummies(titanic.Embarked, prefix='Embarked')\n",
    "embarked_dummies.drop(embarked_dummies.columns[0], axis=1, inplace=True)\n",
    "\n",
    "# concatenate the original DataFrame and the dummy DataFrame\n",
    "titanic = pd.concat([titanic, embarked_dummies], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>Sex_Female</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Embarked_S</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PassengerId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.25</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Survived  Pclass                     Name   Sex   Age  SibSp  \\\n",
       "PassengerId                                                                 \n",
       "1                   0       3  Braund, Mr. Owen Harris  male  22.0      1   \n",
       "\n",
       "             Parch     Ticket  Fare Cabin Embarked  Sex_Female  Embarked_Q  \\\n",
       "PassengerId                                                                  \n",
       "1                0  A/5 21171  7.25   NaN        S           0           0   \n",
       "\n",
       "             Embarked_S  \n",
       "PassengerId              \n",
       "1                     1  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- How do we **interpret** the encoding for Embarked?\n",
    "- Why didn't we just encode Embarked using a **single feature** (C=0, Q=1, S=2)?\n",
    "- Does it matter which category we choose to define as the **baseline**?\n",
    "- Why do we only need **two dummy variables** for Embarked?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7937219730941704\n"
     ]
    }
   ],
   "source": [
    "# define X and y\n",
    "feature_cols = ['Pclass', 'Parch', 'Age', 'Sex_Female', 'Embarked_Q', 'Embarked_S']\n",
    "X = titanic[feature_cols]\n",
    "y = titanic.Survived\n",
    "\n",
    "# train/test split\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)\n",
    "\n",
    "# train a logistic regression model\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "logreg = LogisticRegression(C=1e9)\n",
    "logreg.fit(X_train, y_train)\n",
    "\n",
    "# make predictions for testing set\n",
    "y_pred_class = logreg.predict(X_test)\n",
    "\n",
    "# calculate testing accuracy\n",
    "from sklearn import metrics\n",
    "print(metrics.accuracy_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ROC curves and AUC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# predict probability of survival\n",
    "y_pred_prob = logreg.predict_proba(X_test)[:, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['figure.figsize'] = (8, 6)\n",
    "plt.rcParams['font.size'] = 14"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'True Positive Rate (Sensitivity)')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xm8VXW9//HX+xxAJpkEC5kRFHFWwKnBtEGtq13rmqaV0yUt7ZeWvyz7WdfbaHmbtIG8hpk5VFqUmJU5JwEOKCIIIgqizAIiIIfz+f2xFrE9nLPOOkfW3vuc834+HvvBXtN3ffb3HPbnfL9rre9XEYGZmVlTaiodgJmZVTcnCjMzy+REYWZmmZwozMwskxOFmZllcqIwM7NMhSUKSddJWi5pdhPbJemHkhZIekLSIUXFYmZmrVdki2IycFzG9uOB0elrIvCTAmMxM7NWKixRRMT9wOqMXU4CfhmJaUAfSQOLisfMzFqnUwXPPQhYXLK8JF33UsMdJU0kaXXQo0ePQ8eMGVOWAM3M2oJFqzawflNd5j6vv7xgZUQMaE35lUwUamRdo+OJRMQkYBLAuHHjYubMmUXGZWb2pqxYv5n6Mg6PdOFNj7Fhcx3XfmJck/vs0af7860tv5KJYgkwpGR5MLC0QrGYme0UN01/gS/e9mTZzzthRD8G9u5WSNmVTBRTgAsk3QwcBqyNiB26nczM2pLl6zYD8LUP7keNGus4Kcahw/oWVnZhiULSTcDRQH9JS4CvAJ0BIuKnwFTgBGAB8BpwVlGxmJm9WZu2bOXhZ1fx97nLeWzxGrbWN77fivVJovjohKHU1JQvURSpsEQREac1sz2ATxd1fjOzN2vx6te4d95y/j53Of94dhWb6+rp1rmWccP70rVzbaPHDO7bjT0H9Gw3SQIq2/VkZlZV1r62hZnPr+afz63mnrnLmb/8VQCG7dad0yYM5ZgxuzNhRL8mk0R75URhZh3WsnWbmP7caqY/t5oZi1Yzb9l6IqBzrZgwoh8fGT+EY8bszoj+PVAZrzdUGycKM+sQIoLnVm5gxqLVTH9uDTMWreaF1a8B0L1LLYcM7csJ+w9k/PB+HDSkD926dKxWQxYnCjNrl7bWB0+/tO5frYUZi9aw8tXkQnO/Hl0YN6wvHz9iGOOH92PfPXrRqdZjpDbFicLM2oVNW7byxJK1aYthNY8+v4b1m5OnlQf16cbbR/dn/PB+TBjRlz0H9OzQXUkt5URhZm3Suk1beOT5NcxIWwyzFq/l9fSe1b3e0pMTD9qDCSP6MX54P/boU8yDaB2FE4WZtQkr1m/+V2thxqLVPP3SOuoDOtWIfQf15hNHDmPCiN0YN6wvfXt0qXS47YoThZlVte//7Rn+8PhSnlu5AYBunWs5eGgfLjxmNBNG9OPgoX3o3sVfZUVy7ZpZVfvpfc8ytF93vnTCGMYP78d+g3rT2Reey8qJwsyq3rv23p2J79iz0mF0WE4UZlaVIoIf/X0Bm7bUs1tPX3OoJCcKM6s6W7bWc9ntT3LrzCWcfMggzjxyRKVD6tCcKMysqqzftIVP3fgoD8xfyWeOHc1F7x7tZx4qzInCzKrGy2s3cdbkGTyzbD1XfugAThk/pPmDrHBOFGYGwKKVG5j8j0U89sKaisWweM1GNm/ZynVnjuede7VqemcrgBOFWQcWETy8cBXXPbiIu+cuo1NNMmpqpW4/Hdi7GxceO4p99+hdkfNb45wozDqgTVu2MmXWUq578Dnmvryefj26cOG7RnHG4cPYvVfXSodnVcaJwqwDWb5+E7+a9gI3TnueVRteZ8xbd+XKDx3AiQft0eEm47H8nCjMqlREMPfl9Wyua2Jy5hZ4bXMdv310CX+ctZS6+uDYMbtz9lEjOGLP3XxHkTXLicKsSt0zbzlnT56508rr3qWW0w8bxieOHM6I/j12WrnW/jlRmFWp9ZuSuRS+8e/7M7D3m7tuIMHBQ/vSu1vnnRGadTBOFGattGzdJi686TGeWba+kPJfT7ucjthzN7cArKKcKMxaYd7L6znrF9NZu3EL/37IIGoL6ufv26MLw/p1L6Rss7ycKMxSP7vvWea+3HzrICK4++nldOtSy63nHeF7/q3dc6IwS13112fYpVMNfbs3P1LpfoN6891TDmSQp9i0DsCJwjqMZ1e8yuwX1za5vb4+OP2wYVx6/JgyRmVW/ZworMO4+NZZzFr8SuY+fbv7riCzhpworE2q21rPa1u2tuiYja/XcdSo3bjipP0a3V4j+cKxWSOcKKzNefqldZx7/UxefGVji48dvfuu7DmgZwFRmbVfmYlC0njgDODtwEBgIzAbuAP4dUQUcwO5WRMemL+C83/1KD136cSXThhDTQtvSz16bw9dbdZSTSYKSX8CVgF/AK4ClgNdgb2AdwF3SLoyIv5UjkCtev3h8Rf55cPPF36eiOCJJWsZtXtPfnHWeAb29h1HZuWQ1aI4JyKWNVi3CZievr4taffCIrM24y9PLWPO0nUcOqxv4ef68KGD+dL796FXV190NiuXJhPFtiQh6TzgpojY4b7CiFheYGz2Jt33zAqeXJJ9l8/OMH/5egb17cavzj2s8HOZWfnluZg9HHhU0j+B6yLib8WGZG9WRHDNPQv47l+eKds53zP2LWU7l5mVV7OJIiIulfQl4HjgPEk/AW4iSRqLCo7PGrFwxau89nrTt4b+atrz3DxjMR88aA++efIBdKotfr6BTjWe08Csvcp1e2xE1EtaBCwC9ie5A+oPkqZGxBebOk7SccAPgFrg2oj4VoPtQ4HrgT7pPpdGxNRWfI4O45Hn1/Chn/yj2f0ueNcoPvfevTwpjZm9ac0mCkmfAs4E1gH/C1wWEZsl1QALgEYThaRa4BrgPcASYIakKRExp2S3LwO3RsRPJI0FppJ0dVkT1m3aAsClx49hZBNDTw/YdRcOHlr8hWUz6xjytCgGA6dGxMLSlWkr48SM4yYAC7YdJ+lm4CSgNFEE0Ct93xtYmjfwjuDPs1/mK1NmUx/b121On0Y+fORuHDSkT4UiM7OOJE+i2KNhkpA0OSLOjIjZGccNAhaXLC8BGt4W81XgL5IuBHoA726sIEkTgYkAQ4cOzRFy+zD7xbUsW7eZ0ya88TP36tqJfQbuWqGozKyjyZMoDihdSLucxuc4rrHO8WiwfBowOSKuknQEcIOk/SLiDbPJR8QkYBLAuHHjGpbRrtXWiG+evH+lwzCzDqymqQ2SviBpDXCApNXpaw2wkuRaQnOWAENKlgezY9fSOcCtABHxMMmT3/1bEL+ZmRUsq0VxJcnQHd8ELt22MiLyDtk5AxgtaQTwInAq8NEG+7wAHAtMlrQPSaJYkbP8Nu/1unrue2YFdVvrG92+YPmrZY7IzGxHWYliVETMl3QDsO+2ldtut4yIJ7IKjog6SRcAd5Hc+npdRDwl6QpgZkRMAT4H/FzSRSTdUmdGRIfpWrr2wYVc+ed5mfv069H8bGtmZkXKShSXknQNXdPItgDe0Vzh6TMRUxusu7zk/RzgqFyRtjOb67byi4cWceSeu3H5v41tcr8BPXcpY1RmZjvKGuvpnPTft5cvnI7j94+9yIr1m/neKQcx5q29mj/AzKxCmryYvY2kRyVdImlYOQLqCOrrg0n3L2TswF4cNWq3SodjZpap2UQB/AfQGZgi6WFJn5U0qOC42rW75y7n2RUb+OQ7R3qIDTOres0mioh4NiK+EREHAmcDhwLFz1LTjk26/1kG9enGCfsPrHQoZmbNyjUooKTBwCnAR9JjLisyqPbs0RfWMGPRGv7fB8bSuTZPg87MrLLyDAr4ELAr8BvgYxFRvkkO2qFJ9y2kd7fOnDp+SPM7m5lVgTwtik82M6aT5fTcyg3cNedlPnX0nvTYJVdjzsys4pr8tpJ0WkTcBBwj6ZiG2yPih4VG1g5d+8BCOtfU8Ikjh1c6FDOz3LL+rN02ocGARrZ1mKend5aVr27mt48s4eRDBrH7rl0rHY6ZWW5ZD9z9OH17R0RMK90m6fBCo2qHbpmxmM119Zz79pGVDsXMrEXy3Hbz40bWNTash2V48ZWN9O/ZhVG796x0KGZmLZJ1jWICcAQwQNJnSjb1InkAz1rMD9eZWduTdY2iB8ncEJ1443WK9SRPa5uZWQeQdY3iHuAeSb9oOBWq5RMRTH9uNes31bFkzcZKh2Nm1ipZXU9XRcTngKsk7XCXU0ScXGhk7cAzy17lI5O23wcwckCPCkZjZtY6WV1Pt6T/Xl2OQNqjjVuSyQAv/8BYxg/vxx59fFusmbU9WV1P09N/7962TlJvYFA64ZDlNKJ/D/Yf3LvSYZiZtUqe+SjultRLUl/gSeDXkr5TfGhmZlYN8gw41C8i1kk6B7g+Iv6fpCeASwqOrU2Z+/I6vn3nXOrqt1/OWbeproIRmZntHHkeuOskaQDJLbF/LDieNuvB+Su5Z94K1m3cwobNdWzYXEet4G2j+rPPQE91amZtV54WxdeB+4AHI2K6pJHAc8WG1XbdcO5h9Orq5xHNrP1oNlFExM3AzSXLC4GTigzKzMyqR56Ji/qTTIE6vHT/iJhYXFhmZlYt8nQ9/QGYBjwIbC02HDMzqzZ5EkWP9AltMzPrgPLc9XSnpPcWHomZmVWlPIniPODPkl6VtFrSGkmriw7MzMyqQ56up/6FR9GGXfvAQpas2chTS9dWOhQzs0LkuT12q6RTgZER8Q1Jg4G3AI8UHl2Ve3VzHV+742l26VRD1861jB3Yi+6daysdlpnZTpXn9tirSWa0ewfwDeA14KfA+GJDq34RyXAdl7xvb8+FbWbtVp6upyMj4hBJjwFExGpJXQqOy8zMqkSei9lbJNUAASBpN6C+0KjMzKxq5EkU1wC/AwZI+i+SB+++XWhUbcTsF9cBUCNVOBIzs+LkuZj9S0mPAO9OV/1HRMwuNqzqd8cTL3HRrY8zon8P3n/AwEqHY2ZWmKw5s7sCWyJia0Q8JWkzcDwwEsiVKCQdB/wAqAWujYhvNbLPKcBXSbq2ZkXER1v8Kcpg5aub+cqUp9j4+la2bK3ngfkrOXRYX37+8XH06+FLNmbWfmV1Pd0F7AkgaU9gOjAWuFjS15srWFItSbfV8elxp0ka22Cf0cAXgaMiYl/gs635EOXw5JK13PHESzy/agOvvLaFjx42lBvPPcxJwszavayup34R8Uz6/hPAzRHxKUm7ADOBy5opewKwIB2WHEk3kwxPXjrf9n8C10TEGoCIWN6Kz7DT3PHES6zesLnRbfOWrQfgqlMO4qAhfcoZlplZRWUliih5fwxwFUBEbJaU566nQcDikuUlwGEN9tkLQNJDJN1TX42IPzcsSNJEYCLA0KFDc5y65Za+spFP//rRzH061Yjd3IIwsw4mK1E8JelbwIskX+h/AZDUG8hzm09j+0SD5U7AaOBoYDDwgKT9IuKVNxwUMQmYBDBu3LiGZewUdVuTYv/rxH2bvDjdtXMtPXfJ8+iJmVn7kfWtdy5wETAGOC4iNqTr9wP+J0fZS4AhJcuDgaWN7DMtIrYAz0maR5I4ZuQovxA9d+lE/567VOr0ZmZVp8lEkSaGrzWy/iHgoRxlzwBGSxpB0io5FWh4R9PvgdOAyelMensBC/OFbmZm5dDkXU+Sfi/peEk7JBNJwyRdLunspo6PiDrgApK7p54Gbk1vs71C0onpbncBqyTNAe4BLomIVW/mA7VWfRTSo2Vm1uZldT19GvgccI2kZcAKoCvJcxQvkNyt9LuswiNiKjC1wbrLS94HcHH6qqjZ6TDhw/t3r3AkZmbVJavr6UXSL3FJo4CBwEZgXkSsL1N8ZXPvvBX07taZg4b0rXQoZmZVJdctPBGxAFhQcCwVU18f3PfMCt4+uj+1NR63ycysVJ5BAdu9OS+tY8X6zRy99+6VDsXMrOo4UQD3PbMCgHfuNaDCkZiZVZ9ciUJSl/Q6Rbt077zl7DeoFwN29fMTZmYNNZsoJL0feBL4a7p8kKTbiw6sXNa+toVHnl/D0Xu528nMrDF5WhRXkIzR9ApARDwOtJvWxQMLVlAfcPTe7nYyM2tMnruetkTEK3rjLG5t+um0F1/ZyLfunMvrdVuZv+zV9LZYjwhrZtaYPIni6XRyoZp0OI7/A0wrNqxiTXt2FX+ctZSR/XvQpVMNE98xkk61vq5vZtaYPIniAuByoB64jWTYjS8WGVS5TD5rAkN385PYZmZZ8iSK90XEF4AvbFsh6WSSpGFmZu1cnv6WLzeyrrnZ7czMrJ1oskUh6X3AccAgSaXzT/Qi6YYyM7MOIKvraTkwG9gEPFWyfj1waZFBmZlZ9cgaPfYx4DFJN0bEpjLGZGZmVSTPxexBkr4OjCWZjwKAiNirsKjMzKxq5EkUk0mmRP0ucDxwFm30GsVtjy5h/vJXmfdyu5tOw8ysMHkSRfeIuEvSdyPiWeDLkh4oOrAiXPq7J6mrr6dTTQ0De3elb4/OlQ7JzKzq5UkUm5WM3/GspPOAF4E2OYJefQTnH70nl7xvTKVDMTNrM/IkiouAnsBngK8DvYGziwzKzMyqR7OJIiL+mb5dD3wMQNLgIoMyM7PqkZkoJI0HBgEPRsRKSfuSDOVxDNAmksUN057n23fOJSKoqw9q5DmxzcxaIuvJ7G8CHwJmkVzAvp1k5NhvA+eVJ7w3b87SdWytD04/bCg1NeLkQ9pEfjMzqxpZLYqTgAMjYqOkfsDSdHleeUJrvZfWbuQHf5vP+s11PLlkLT27duLLHxhb6bDMzNqkrESxKSI2AkTEaklz20KSmLN0HWdPnsHajVvYo09XOteKo0a9pdJhmZm1WVmJYqSkbUOJCxheskxEnFxoZK3wjwUrmXjDI/TcpRO3fepI9hnYq9IhmZm1eVmJ4kMNlq8uMpCd4Zt3zqVfjy7c8snDGdi7W6XDMTNrF7IGBby7nIHsDJvrtrLvHr2cJMzMdiJPFG1mZpmcKMzMLFPuRCFplyIDMTOz6tRsopA0QdKTwPx0+UBJPyo8MjMzqwp5WhQ/BD4ArAKIiFnAu4oMyszMqkeeRFETEc83WLe1iGDMzKz65BlmfLGkCUBIqgUuBJ4pNiwzM6sWeVoU5wMXA0OBZcDh6bpmSTpO0jxJCyRdmrHfhyWFpHF5yjUzs/LJ06Koi4hTW1pw2vq4BngPsASYIWlKRMxpsN+uJJMi/XPHUszMrNLyJIoZkuYBtwC3RcT6nGVPABZExEIASTeTjEg7p8F+/w1cCXw+Z7lv8MD8FVx0y+PU1QfrNm5h1O49W1OMmZk1odmup4jYE/gacCjwpKTfS8rTwhgELC5ZXpKu+xdJBwNDIuJPWQVJmihppqSZK1aseMO2eS+vZ+Wrr3P8fgP52OHDOPuoETlCMzOzvPK0KIiIfwD/kPRV4PvAjcDNzRzW2FRy8a+NUg3wPeDMHOefBEwCGDduXDS2z5dOGMOuXTs3V5SZmbVQngfueko6XdIfgenACuDIHGUvAYaULA8mmfxom12B/YB7JS0iuUg+xRe0zcyqS54WxWzgj8CVEfFAC8qeAYyWNAJ4ETgV+Oi2jRGxFui/bVnSvcDnI2JmC85hZmYFy5MoRkZEfUsLjog6SRcAdwG1wHUR8ZSkK4CZETGlpWWamVn5NZkoJF0VEZ8Dfidph+sCeWa4i4ipwNQG6y5vYt+jm422xOt19Wyq28qmLX5I3MysSFktilvSf6tuZrvX6+o54pt3s2rD6/9aV1vT2LVzMzN7s7JmuJuevt0nIt6QLNIupYrNgLepbiurNrzOu/d5C4eP7MegPt3o3iXXDVxmZtZCeYbwOLuRdefs7EBa4/CR/Tj37SM5fv+BlQ7FzKzdyrpG8RGSO5VGSLqtZNOuwCtFB2ZmZtUhq79mOskcFINJxmzaZj3wWJFBmZlZ9ci6RvEc8Bzwt/KFY2Zm1Sar6+m+iHinpDWUDL1BMjRHRES/wqMzM7OKy+p62jbdaf+MfczMrJ1r8q6nkqexhwC1EbEVOAL4JNCjDLGZmVkVyHN77O9JpkHdE/glsA/w60KjMjOzqpEnUdRHxBbgZOD7EXEhDeaVMDOz9itPoqiT9B/Ax4BtEwx54gczsw4i75PZ7yIZZnxhOmz4TcWGZWZm1aLZAZIiYrakzwCjJI0hmQf768WHZmZm1aDZRCHp7cANJJMPCXirpI9FxENFB2dmZpWXZ8jV7wEnRMQcAEn7kCQOT1lqZtYB5LlG0WVbkgCIiKeBLsWFZGZm1SRPi+JRST8jaUUAnI4HBTQz6zDyJIrzgM8A/5fkGsX9wI+KDMrMzKpHZqKQtD+wJ3B7RFxZnpDMzKyaNHmNQtKXSIbvOB34q6TGZrozM7N2LqtFcTpwQERskDQAmApcV56wzMysWmTd9bQ5IjYARMSKZvY1M7N2KqtFMbJkrmwBe5bOnR0RJxcamZmZVYWsRPGhBstXFxmImZlVp6w5s+8uZyBmZladfN3BzMwyOVGYmVmm3IlC0i5FBmJmZtWp2UQhaYKkJ4H56fKBkjyEh5lZB5GnRfFD4APAKoCImEUy452ZmXUAeRJFTUQ832Dd1iKCMTOz6pNn9NjFkiYAIakWuBB4ptiwzMysWuRpUZwPXAwMBZYBh6frzMysA2g2UUTE8og4NSL6p69TI2JlnsIlHSdpnqQFki5tZPvFkuZIekLS3ZKGteZDmJlZcZrtepL0cyAaro+Iic0cVwtcA7wHWALMkDSldFpVkpnyxkXEa5LOB64EPtKC+M3MrGB5rlH8reR9V+DfgcU5jpsALIiIhQCSbgZOAkrn376nZP9pwBk5yjUzszJqNlFExC2ly5JuAP6ao+xBvDGhLAEOy9j/HODOxjZImghMBBg6dGiOU5uZ2c7SmiE8RgB5riWokXU7dGEBSDoDGAd8p7HtETEpIsZFxLgBAwbkDtTMzN68PNco1rD9C74GWA3scGG6EUuAISXLg4GljZT/buAy4J0RsTlHuWZmVkaZiUKSgAOBF9NV9RHRaKugETOA0ZJGpMefCny0QfkHAz8DjouI5S0J3MzMyiOz6ylNCrdHxNb0lTdJEBF1wAXAXcDTwK0R8ZSkKySdmO72HaAn8BtJj0ua0rqPYWZmRclz19N0SYdExKMtLTwipgJTG6y7vOT9u1tappmZlVeTiUJSp7RV8DbgPyU9C2wguUgdEXFImWJ8g6WvbOSbU+dW4tRmZh1SVotiOnAI8MEyxZLLqg2vM/XJl+jfswtj3tqr0uGYmbV7WYlCABHxbJliyaVrp1pmfeW9lQ7DzKzDyEoUAyRd3NTGiPifAuIxM7Mqk5UoaknuSGrswTkzM+sgshLFSxFxRdkiMTOzqpT1HIVbEmZmlpkoji1bFGZmVrWaTBQRsbqcgZiZWXVqzeixZmbWgThRmJlZJicKMzPL5ERhZmaZnCjMzCyTE4WZmWVyojAzs0xOFGZmlsmJwszMMjlRmJlZJicKMzPL5ERhZmaZnCjMzCyTE4WZmWVyojAzs0xOFGZmlsmJwszMMjlRmJlZJicKMzPL5ERhZmaZnCjMzCyTE4WZmWVyojAzs0xOFGZmlsmJwszMMjlRmJlZpkIThaTjJM2TtEDSpY1s30XSLen2f0oaXmQ8ZmbWcoUlCkm1wDXA8cBY4DRJYxvsdg6wJiJGAd8Dvt1cuYP6dtvZoZqZWYYiWxQTgAURsTAiXgduBk5qsM9JwPXp+98Cx0pSVqHdu9Tu9EDNzKxpnQosexCwuGR5CXBYU/tERJ2ktcBuwMrSnSRNBCami5slzS4k4ranPw3qqgNzXWznutjOdbHd3q09sMhE0VjLIFqxDxExCZgEIGlmRIx78+G1fa6L7VwX27kutnNdbCdpZmuPLbLraQkwpGR5MLC0qX0kdQJ6A6sLjMnMzFqoyEQxAxgtaYSkLsCpwJQG+0wBPpG+/zDw94jYoUVhZmaVU1jXU3rN4QLgLqAWuC4inpJ0BTAzIqYA/wvcIGkBSUvi1BxFTyoq5jbIdbGd62I718V2rovtWl0X8h/wZmaWxU9mm5lZJicKMzPLVLWJwsN/bJejLi6WNEfSE5LuljSsEnGWQ3N1UbLfhyWFpHZ7a2SeupB0Svq78ZSkX5c7xnLJ8X9kqKR7JD2W/j85oRJxFk3SdZKWN/WsmRI/TOvpCUmH5Co4IqruRXLx+1lgJNAFmAWMbbDPp4Cfpu9PBW6pdNwVrIt3Ad3T9+d35LpI99sVuB+YBoyrdNwV/L0YDTwG9E2Xd6903BWsi0nA+en7scCiSsddUF28AzgEmN3E9hOAO0meYTsc+Geecqu1RVHI8B9tVLN1ERH3RMRr6eI0kmdW2qM8vxcA/w1cCWwqZ3Bllqcu/hO4JiLWAETE8jLHWC556iKAXun73uz4TFe7EBH3k/0s2knALyMxDegjaWBz5VZromhs+I9BTe0TEXXAtuE/2ps8dVHqHJK/GNqjZutC0sHAkIj4UzkDq4A8vxd7AXtJekjSNEnHlS268spTF18FzpC0BJgKXFie0KpOS79PgGKH8HgzdtrwH+1A7s8p6QxgHPDOQiOqnMy6kFRDMgrxmeUKqILy/F50Iul+OpqklfmApP0i4pWCYyu3PHVxGjA5Iq6SdATJ81v7RUR98eFVlVZ9b1Zri8LDf2yXpy6Q9G7gMuDEiNhcptjKrbm62BXYD7hX0iKSPtgp7fSCdt7/I3+IiC0R8RwwjyRxtDd56uIc4FaAiHgY6EoyYGBHk+v7pKFqTRQe/mO7Zusi7W75GUmSaK/90NBMXUTE2ojoHxHDI2I4yfWaEyOi1YOhVbE8/0d+T3KjA5L6k3RFLSxrlOWRpy5eAI4FkLQPSaJYUdYoq8MU4OPp3U+HA2sj4qXmDqrKrqcobviPNidnXXwH6An8Jr2e/0JEnFixoAuSsy46hJx1cRfwXklzgK3AJRGxqnJRFyNnXXwO+Lmki0i6Ws5sj39YSrqJpKuxf3o95itAZ4CI+CnJ9ZkTgAXAa8BZucpth3VlZmY7UbV2PZmZWZVwojAzs0xOFGZmlsmJwszMMjlRmJlZJieKDkrSVkmPl7yGZ+w7vKnRKFt4znvTET5npcNK7N2KMs72a0i3AAAGAElEQVST9PH0/ZmS9ijZdq2ksTs5zhmSDspxzGcldW/Fub4v6R3p+wvSUT0jfe6hpWXtncb+uKSnJe3U2d0knbhtZFZJA5SM2vyYpLdLmiqpT8axTf7cMo75m6S+O+8TWKtVerRDvyrzAl5twb7DaWI0yhae817S0VyBicCUnVXeTq6b0jjPAv6a45hFQP8WnqcfMK1k+eC0rltcVnr8XcBJJcv7F/j7cypwfZE/N5IHai8r6jP4lf/lFoX9S9pyeEDSo+nryEb22VfS9PSv1ickjU7Xn1Gy/meSaps53f3AqPTYY9O/TJ9UMp7+Lun6b2n7PBvfTdd9VdLnJX2YZFyrG9Nzdkv/mh4n6XxJV5bEfKakH7UyzocpGTRN0k8kzVQyv8N/pes+A+wB3CPpnnTdeyU9nNbjbyT1bKTsDwN/3rYQEY9FxKJm4skykGSIhm3lPZnGcqakP0j6c9pS+krJ52m0PpTM7/Bo2qq6u6Scq9MW1pXACSV1v2hbK0jSx9Of2SxJN6Trmvq5vV/S7SXxvEfSbeniFJIxmqzSKp2p/KrMi+RJ3cfT1+3puu5A1/T9aJKnWqGkRQH8CDg9fd8F6AbsA/wR6Jyu/zHw8UbOeS/b/1K/BLiFZCiFxcBe6fpfAp8l+Wt7HtsfCu2T/vtV4PMNyytdBgaQDDu9bf2dwNtaGedngW+UbOuX/lub7ndAuryItBVAMobQ/UCPdPkLwOWNnOd64N8aWf+vslr4Mz2LZBTlO4GLSursTOAlktGVuwGz03pqtD7S+lsMjGjwmc8Erm74vjRmYN/059a/wbGN/txIBqmbCwxIl39dWifAfGC3Sv9/6eivqhzCw8piY0Q07HvvDGz7i3ErydhADT0MXCZpMHBbRMyXdCxwKDBDyRAi3YCmxpy6UdJGki+WC4G9geci4pl0+/XAp4GrSeaTuFbSHUDuYcMjYoWkhUrGspmfnuOhtNyWxNmDJCGUzgJ2iqSJJMPfDCSZBOeJBscenq5/KD1PF5J6a2ggO3G8oYj4haS7gONI5h34pKQD081/jXT4jvQv9rcBdTReH4cD90cykCAR0ZLBNo8BfhsRK/McGxGRtjrOkPQL4AiSZLXNcpLWWrsbeqQtcaKwUhcBy4ADSW502GHin4j4taR/Au8H7pJ0LslfhddHxBdznOP0KBmkT1Kjc4hEMn7PBJKB3E4FLiD5EsrrFuAUkr9Wb0+/kFoUJ8lMad8CrgFOljQC+DwwPiLWSJpM0iJqSCRfzM11m2xs4vgmpV+mBwNLI2KH6TwjYilwHXCdkhsQ9tu2qeGuNPFzk3RiI/vnDrEVx/6CpGWzCfhNJPPLbNOVpJ6sgnyNwkr1Bl6KZIz+j5H8Nf0GkkYCCyPihyR9yAcAdwMflrR7uk8/5Z+3ey4wXNKodPljwH1pn37viJhK0v3T2J1H60mGFm/MbcAHSfq4b0nXtSjOiNgCfBk4XMmIo72ADcBaSW8Bjm8ilmnAUds+k6TukhprnT1Nep0mr4g4KyIOaixJpNcVOqfv30rS1fRiuvk96eftRlIvD9F0fTwMvDNNjEjq14IQ7yZpde2Wcewbfm5pcltKUteTSz6PgLeStD6tgpworNSPgU9ImkbS7bShkX0+AsyW9DgwhmRaxTkk/8n/IukJ4K8k3SrNiohNJH3rv5H0JFAP/JTki+RPaXn3kbR2GpoM/HTbBdUG5a4B5gDDImJ6uq7FcUbERuAqkv71WSRzUD9F8lf7QyW7TgLulHRPRKwg6cO/KT3PNJK6augOkpE+geSiuJIRPwcDT0i6Niu2RryX5Gczi+QOqEsi4uV024PADSTXpH4XETObqo80/onAbWlZtzQ8UVMi4ing6yTJfhbwP43sNpkdf243AovTmLY5lOSusLqGBVh5efRYswqS9CDwgShw1jlJZ5JcPL6gqHO8WZKuBh6LiP8tWfcDkluo765cZAZuUZhV2ueAoZUOopIkPULShfmrBptmO0lUB7cozMwsk1sUZmaWyYnCzMwyOVGYmVkmJwozM8vkRGFmZpn+PxvEdstpBSQuAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f45862cc128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot ROC curve\n",
    "fpr, tpr, thresholds = metrics.roc_curve(y_test, y_pred_prob)\n",
    "plt.plot(fpr, tpr)\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.0])\n",
    "plt.xlabel('False Positive Rate (1 - Specificity)')\n",
    "plt.ylabel('True Positive Rate (Sensitivity)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8386924342105262\n"
     ]
    }
   ],
   "source": [
    "# calculate AUC\n",
    "print(metrics.roc_auc_score(y_test, y_pred_prob))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides allowing you to calculate AUC, seeing the ROC curve can help you to choose a threshold that **balances sensitivity and specificity** in a way that makes sense for the particular context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([<matplotlib.axes._subplots.AxesSubplot object at 0x7f4586279780>,\n",
       "       <matplotlib.axes._subplots.AxesSubplot object at 0x7f45809a6ef0>],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAETCAYAAADecgZGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAEO5JREFUeJzt3X+w5XVdx/Hny10Iwx+ArLiBt2VqLbUZ0AxpmGlUSlFM+ENKzVgnZrYpbTQbdTWn1KzAZjKbsR8MkJulwJgIifmjFSzH5DeYuMkq4RVBFxP8NakB7/4435XLcvfec+8953vOuZ/nY+bO/Z7v+R7O637vfl587/d8zvekqpAkteVhkw4gSeqf5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlPmSRHJLkkyXeTfCnJSyadSZqEJK9Icm2S7yd516TzrDcbJx1AD/FO4AfAUcDxwOVJbqqqmycbS+rdHcBbgecAD59wlnUnvsN3eiQ5FLgb+JmquqVb927gK1W1Y6LhpAlJ8lbgmKp62aSzrCee9pkuTwDu21f8nZuAJ08oj6R1yvKfLo8Avrnfum8Cj5xAFknrmOU/Xb4DPGq/dY8Cvj2BLJLWMct/utwCbEyydcG64wBf7JU0Upb/FKmq7wLvB96S5NAkJwGnAe+ebDKpf0k2JjkE2ABsSHJIEmcojojlP31+m8G0tr3Ae4HfcpqnGvVG4H+BHcBLu+U3TjTROuJUT0lqkEf+ktQgy1+SGmT5S1KDLH9JapDlL0kN6nXO7JFHHllbtmzp8ynVgOuuu+7rVbVp0jlWwrGgcVjJWOi1/Lds2cK1117b51OqAUm+NOkMK+VY0DisZCx42keSGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUoKn5VJwtOy5f8WNuO/vUMSSRpPVvqPJPchuDDxG/D7i3qp6W5AjgImALcBvwK1V193hiSpJGaSWnfZ5ZVcdX1dO62zuAXVW1FdjV3ZYkzYC1nPM/DdjZLe8ETl97HElSH4Yt/wI+muS6JNu7dUdV1Z0A3ffHLvbAJNuT7Ely1/z8/NoTSzPKsaBpMmz5n1RVTwWeC7w8yS8M+wRVdW5Vba2qTXNzc6sKKa0HjgVNk6HKv6ru6L7vBS4BTgC+lmQzQPd977hCSpJGa9nyT3JokkfuWwaeDXwWuAzY1m22Dbh0XCElSaM1zFTPo4BLkuzb/j1V9eEk1wAXJzkLmAfOGF9MSdIoLVv+VXUrcNwi6/8HOHkcoSRJ4+XlHSSpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JatDQ5Z9kQ5Ibknywu31skquS7ElyUZKDxxdTkjRKKznyfyWwe8Htc4C3V9VW4G7grFEGkySNz1Dln+QY4FTgvO52gGcB7+s22QmcPo6AkqTRG/bI/y+A1wL3d7cfA9xTVfd2t28Hjl7sgUm2d6eG7pqfn19TWGmWORY0TZYt/yTPB/ZW1XULVy+yaS32+Ko6t6q2VtWmubm5VcaUZp9jQdNk4xDbnAS8IMnzgEOARzH4S+CwJBu7o/9jgDvGF1OSNErLHvlX1eur6piq2gK8CPh4Vf0acAXwwm6zbcClY0spSRqptczzfx3w6iRfYPAawPmjiSRJGrdhTvv8UFVdCVzZLd8KnDD6SJKkcfMdvpLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNWjZ8k9ySJKrk9yU5OYkb+7WH5vkqiR7klyU5ODxx5UkjcIwR/7fB55VVccBxwOnJDkROAd4e1VtBe4GzhpfTEnSKC1b/jXwne7mQd1XAc8C3tet3wmcPpaEkqSRG+qcf5INSW4E9gIfA74I3FNV93ab3A4cPZ6IkqRRG6r8q+q+qjoeOAY4AXjiYpst9tgk27vXBe6an59ffVJpxjkWNE1WNNunqu4BrgROBA5LsrG76xjgjgM85tyq2lpVm+bm5taSVZppjgVNk2Fm+2xKcli3/HDgF4HdwBXAC7vNtgGXjiukJGm0Ni6/CZuBnUk2MPifxcVV9cEknwMuTPJW4Abg/DHmlCSN0LLlX1WfAZ6yyPpbGZz/lyTNGN/hK0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhpk+UtSg4Z5k9fU2rLj8hVtf9vZp44piSTNFo/8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWrQxkkHkKRWbdlx+aoed9vZp675uZc98k/y+CRXJNmd5OYkr+zWH5HkY0n2dN8PX3MaSVIvhjntcy/we1X1ROBE4OVJngTsAHZV1VZgV3dbkjQDli3/qrqzqq7vlr8N7AaOBk4Ddnab7QROH1dISdJoregF3yRbgKcAVwFHVdWdMPgfBPDYAzxme3dq6K75+fm1pZVmmGNB02To8k/yCOCfgFdV1beGfVxVnVtVW6tq09zc3GoySuuCY0HTZKjyT3IQg+L/x6p6f7f6a0k2d/dvBvaOJ6IkadSGme0T4Hxgd1X9+YK7LgO2dcvbgEtHH0+SNA7DzPM/Cfh14D+T3NitewNwNnBxkrOAeeCM8UQcndXMqR3FfFpJmjbLln9VfRLIAe4+ebRxJEl98PIOktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUFez1+SRmC11+afFI/8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAb5Gb7SFFvN58LedvapY0ii9cYjf0lqkOUvSQ2y/CWpQZa/JDVo2fJPckGSvUk+u2DdEUk+lmRP9/3w8caUJI3SMEf+7wJO2W/dDmBXVW0FdnW3JUkzYtnyr6p/A76x3+rTgJ3d8k7g9BHnkiSN0Wrn+R9VVXcCVNWdSR57oA2TbAdeAxy2adOmVT6dNPscC/3zfRIHNvYXfKvq3KraWlWb5ubmxv100tRyLGiarLb8v5ZkM0D3fe/oIkmSxm215X8ZsK1b3gZcOpo4kqQ+LHvOP8l7gWcARya5HfhD4Gzg4iRnAfPAGeMMKWn9WM15eOjvXPxq882aZcu/ql58gLtOHnEWSVJPfIevJDXI8pekBnk9/2U4T1iaDq2ci++LR/6S1CDLX5IaZPlLUoM85z/DfD1C0mp55C9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoOc5z8Gzr+XNO088pekBln+ktQgy1+SGuQ5/ynhtco1Kr7mpGF45C9JDbL8JalBlr8kNchz/o3p67UFzyG3wdeqZpdH/pLUIMtfkhpk+UtSgzznL8lz9w3yyF+SGmT5S1KDLH9JapDlL0kNWtMLvklOAd4BbADOq6qzR5JKM8+Li0nTbdVH/kk2AO8Engs8CXhxkieNKpgkaXzWctrnBOALVXVrVf0AuBA4bTSxJEnjtJbyPxr48oLbt3frJElTbi3n/LPIunrIRsl24DXAYcD3ktzc3XUk8PU1PP+omWdpY8+Tc1a0+cI8Pz7yMGOwxFiABn/fK2SeBRYZK/vyDD0WUvWQvh7ugcnPA2+qqud0t18PUFV/OuTjr62qp63qycfAPEszz3hN289jnqWthzxrOe1zDbA1ybFJDgZeBFy2hv+eJKknqz7tU1X3JnkF8BEGUz0vqKqbl3mYJGkKrGmef1V9CPjQKh9+7lqeewzMszTzjNe0/TzmWdrM51n1OX9J0uzy8g6S1CDLX5IaZPlLUoN6+ySvJGFwSYijGbwZ7A7g6pqCFx2SHAs8BfhcVf3XhDI8GjiFB++fj1TVPZPIs9A07J/1xLGwbAbHQg96OfJP8mxgD/Am4HnAqcCbgT3dfb1K8oEFy6cBHwd+Gbg0ycsmkOdM4HrgGcCPAocCzwSu6+7rO89U7Z8ux6OT/GqSVyf53W75sElkWQvHwrJ5HAvD5Vr7eKiqsX8Bu4Eti6w/FtjdR4b9nveGBcufAo7tlo8EbppAns8Dhy2y/nDgFvcPZwJfBP4aeGP39TfdujP7zrPGn8WxsHQex8LymUYyHvo67bORwYXf9vcV4KCeMiy08M/rjVX13wBV9fUk908gT1jkukjA/Sx+DaVxm7b98/vAz9Z+f/YnORy4Cvj7CWRaLcfC0hwLyxvJeOir/C8ArklyIQ9cCfTxDC4JcX5PGRY6Lsm3GPxj+pEkj6uqr3aXqdgwgTx/DFyf5KM8sH/mgF8C/mgCeaZt/0xbIayFY2FpjoXljWQ89PYmr+6DXl7A4EWcMDj6uayqPtdLgCF058yeWFX/MYHnPhx4Dg/ePx+pqrv7znIgk9o/SbYBfwAsWghV9a4+86yVY2HZ53YsLP3cIxkPvb/DN8kRQE3TL1LTbxYKYaUcC1qtUYyHvmb7zCW5MMleBuekrk6yt1u3pY8M++V5fPfc/57kDUkOWnDfB5Z67Jjy/MaC5aOT7Epyd5JPJXnCBPJ8I8l5SU7upiVOXPeP+oruaxdwxSyWpmNh2TyOhSGMYjz09Savi4BLgM1VtbWqtgKbgQ8w+PjHvl0AXAn8TpfjE0ke0903iQ8GecWC5bcDFwOPAf6MwSv6fbsLuBF4C3B7knckOXECOQBIcnySTzP4nZ3DYL98Ismnkzx1UrlWybGwNMfCMkY2HnqamrRnNfeNMc+N+91+KXAz8BPA9RPIc/2C5f2z3dBnlkXyzAGvZTD3+lbgTybx+wKevsj6E5nQdLs1/CyOhaXzOBaG+J2NYjz0NdvnuiR/BezkwTMctgE39JRhoYOSHFJV3wOoqn9I8lUGn01w6ATyHJPkLxmcu9uU5KCq+r99WSeQ54d/3lbVPPA24G1JforBrJS+HVpVV+2/sqo+nWQSv6+1cCwszbGwvJGMh77K/0zgLAbvZNz3AsWXgX9mMtPbzgOeDnxi34qq+tckZzD45fbtNQuWrwUeAdyd5HFM5tPRrlhsZVV9nsHvsG//kuRyBvOXFxbmmcCHJ5BnLRwLS3MsLG8k48Hr+WsmJHkucBoPnR652g8TkmbWKMbDxMs/yfOr6oMTDbGAeZY2bXnWk2nbt+ZZ2rTlWalpuKTzz006wH7Ms7SpypNk+6QzjNBU7VvMs5xpy7Oi8dDnO3x/mgf+TNl3mdbLqmp3LwHMM9N5DiTJb1bV3046x0pM2741z2zlWcpKxkNfb/J6HYM5zAGuBq7plt+bZEcfGcwzu3mW8YNJB1iJadu35pmtPEMYejz0cuSf5BbgyQumbO1bfzBwcw3e6NIb88xWnqUkma+quUnnGNa07VvzzFae5axkPPQ11fN+4MeAL+23fnN3X9/Ms7SpypPkMwe6CziqzywjMFX7FvMsZ9ryjGw89FX+rwJ2JdnDg69C95M8+O3cfTHPbOU5isFFrPa/dkkYfMDGLJm2fWue2coDIxoPfb7g+zAe+NzSffNSr6mq+3oJYJ6ZzZPkfODvquqTi9z3nqp6Sd+Z1mKa9q15ZjLPSMbDxOf5S5L6Nw3z/CVJPbP8JalBlr8kNcjyl6QGWf6S1KD/B3xbEU9laU1VAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f4586549fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# histogram of predicted probabilities grouped by actual response value\n",
    "df = pd.DataFrame({'probability':y_pred_prob, 'actual':y_test})\n",
    "df.hist(column='probability', by='actual', sharex=True, sharey=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f45808ab320>]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAGrhJREFUeJzt3WtsXPd55/Hvo/udGoqUowsvsi1ZUhRZJseys3Fs2YoDO23tLpC2Uje7m643RtK6C2yKBVxkkS3cN7spukELGE3VNkhboHXdvmiFQoWLirS9dazUpCQ7ljSSaIU3SdZQ4pC6kBRvz76YITWkSPGIGs7lnN8HEMCZOSKfI1I/Hf3nOf/H3B0REQmXeYUuQEREck/hLiISQgp3EZEQUriLiISQwl1EJIQU7iIiIaRwFxEJIYW7iEgIKdxFREJoQaG+cEVFhdfW1hbqy4uIlKTm5ubL7l4503EFC/fa2lqampoK9eVFREqSmbUFOU7LMiIiIaRwFxEJIYW7iEgIKdxFREJI4S4iEkIzhruZ/dDMkmb28TSvm5n9oZm1mNlHZlaX+zJFRORuBLly/xHw3B1efx7YnPn1MvBH916WiIjcixnD3d3fBbrvcMiLwF942hFgtZmty1WBIiJhcP3mMP969jJ/8C9n+fh875x/vVzcxLQB6Mh63Jl57uLkA83sZdJX91RXV+fgS4uIFB9353xPP81tKZrbUjS1pkh8epVRBzMoX7GIHRvK5rSGXIS7TfHclFO33f0AcAAgHo9rMreIhMLQyCgnL1ylqS3F0bYUTW3dXLp6E4Bli+bzSPVqXnlmM/U1MR6pXs2qJQvnvKZchHsnUJX1eCNwIQefV0SkKPX0DXK0PX1F3tyW4sPOHgaGRgHYsHopj21aQ31NjPqaGFs/s5IF8/PfmJiLcD8IvGJmbwCPAb3uftuSjIhIKXJ3fnb5RtZVeYqW5HUA5s8zPrt+Fft3V4+H+bqypQWuOG3GcDezvwb2ABVm1gn8L2AhgLv/ADgEfAVoAfqAX5urYkVE5trA0Ag/Pd87flV+tD1F941BAFYtWUB9TYx//8gG6qpjPFxVxrJFBdt/8Y5mrMrd98/wugO/kbOKRETyKHltIH1F3pqiuT3Fx+d7GRpJvyV4f8Vyntm6lnjmqvyByhXMmzfV24zFpzj/yRERmQMjo86ZS9fGu1ia21K0d/cBsGjBPB7eWMZ/eWIT8Zpy6qpXs2bF4gJXPHsKdxEJres3h/mwo4em1nQHy/H2Hq7dHAagYsUi6mti/MfHa6iribFjwyoWL5hf4IpzR+EuIqEwube8uS3FqYu3essfum8lv7Br/fgSS3X5MsxKY4llNhTuIlKSxnrLs8P806sDQFZv+dMPUl9bzq6q1ZQtnfve8mKicBeRkjDWWz52x+fk3vLdm8oL3lteTBTuIlJ0gvSW73u0mnhtcfWWFxOFu4gU3Fhv+dhV+VS95b+4az31NeVF3VteTPQnJCJ513XtJs1t3ekwb5vYW74p01teXxMjXmK95cVE4S4ic2pk1DmbvJa+Is+EeXZv+c4N6d7y+ur0Eksp95YXE4W7iORUdm95c3uKY22pyPSWFxOFu4jMmrtzoXeAptbu8avy7N7yLWvTveX11THiteHvLS8mCncRCWxoZJRTF6+OX5U3t07sLd9Vle4tr6uJ8Uh1LHK95cVE4S4i0+rtG0rvW5558/PDjl76h0aAdG/5o5vKx+/4VG95cVG4iwhwq7c8+47Ps5N6y3/l0Sr1lpcIhbtIRA0MjfDx+V6aMkF+tC3Flaze8rqaGC/uWk9dTYxdVavVW15i9N0SiYh0b3lqvL/84/NXGRxJ376/qWI5ex5aO35V/qB6y0uewl0khEZHnTPJzL7lmTc/265kesvnz2PnxjJ+7Qu11NfEqKuJUaHe8tBRuIuEwI2bwxzv6Bm/4/NYe4prAxN7y//DY9XU15SrtzwiFO4iJWh83/LW7ql7yx9Wb3nUKdxFitxYb/nYVfnRthQXeyf2lv/G0w9Sr95yyaJwFykyY73l6TDvntBbvr5sCfHacuqrVxOvLVdvuUxL4S5SQO5O65W+8S6WptaJveXb16V7y8eGUKxfrd5yCUbhLpJHY73l2Uss6i2XuaCfHJE5NNZbfrQ9RVNr94Te8to1y9jzUGbf8lr1lktuKdxFcmR01DmbvD6+D0tz28Te8s9lesvrMkss6i2XuaRwF5mlG2P7lo/dvp/VW75meXZveYwdG8rUWy55pXAXCehCT3/WwOZuTl28xsioj/eW//zO9eM7JNasUW+5FJbCXWQKwyOjnLp4bcISy+Te8l/f84B6y6VoKdxFyPSWd2T2YWlLcbyjZ0Jv+diw5vqacratU2+5FD+Fu0SOu9N2pW98rby5rZszl9RbLuESKNzN7DngD4D5wJ+6+/+e9Ho18OfA6swxr7r7oRzXKjIrA0MjnLjQmx4Nl3nj8/L1dG/5yiULqK+J8Qs711NfG+PhjatZvljXPFL6ZvwpNrP5wOvAs0An8IGZHXT3k1mH/U/gTXf/IzPbDhwCauegXpEZXb5+c8I0oZ929k7oLX9qi3rLJfyCXKLsBlrc/RyAmb0BvAhkh7sDqzIflwEXclmkyHTGesvH9mE52paidVJv+dfH9i2vjlG5Ur3lEg1Bwn0D0JH1uBN4bNIxvwP8s5n9JrAc+FJOqhOZpG8ws295ZgDF0bYUVyf1lu/fXU28NsZn15exZKF6yyWagoT7VP9n9UmP9wM/cvffN7PPA39pZjvcfXTCJzJ7GXgZoLq6ejb1SsRcGNu3PPPr5MWrjIymf/y23LeCn9u5fryTRb3lIrcECfdOoCrr8UZuX3Z5CXgOwN3fN7MlQAWQzD7I3Q8ABwDi8fjkfyAk4sZ6y5vbusdvFrqQ6S1fuvBWb3ldTYy6qhhly9RbLjKdIOH+AbDZzDYB54F9wK9OOqYd2Av8yMy2AUuArlwWKuHT25/et/xoW4qm1om95esyveUvq7dcZFZmDHd3HzazV4C3SLc5/tDdT5jZa0CTux8Efgv4EzP776SXbL7u7royl3FjveXZW92eSV7DPd1bvm3dSn7l0SrqMkss6i0XuTdWqAyOx+Pe1NRUkK8tc+/mcNa+5a2395bXVcfG92F5uEq95SJBmVmzu8dnOk5/oyQnxnrLj2auzCf3lj+5pZJ4TTn1NTE2r1VvuchcU7jLXRsddVq6ro/f8dnc1q3ecpEio3CXGY31lh/NWi/P7i2vy/SWj+1brt5ykcJTuMttLvb2Z12VT9Vbvo76zBJLrXrLRYqSwj3ihkdGSXx6jabWbprbe2hu7b6tt/xbTz1Afa16y0VKicI9Ynr7hzjWfuuq/HhHD32DE3vLv1ETI15TztZ1K1mo3nKRkqRwDzF3p727j6bW6XvLf6l+I/W15eotFwkZhXuIpHvLr9I8Phquh8vXbwK3est/buc64uotFwk9/e0uYVcm7Vv+0fleBofTveU1a5bx5JaKzKZa5eotF4kYhXuJGOstz77j82eXbwDp3vIdG1bx9X9XS111+q5P9ZaLRJvCvUj1DQ7zYUfvhB0SJ/eW78vM+VRvuYhMpnAvEhd7+ydclZ+4cKu3fPPadG95XXWMeG25estFZEYK9wIY6y3P3iHxfE8/AEsWzrvVW565fV+95SJytxTueXB1YIijWZtqZfeWf2bVEuprY/zXL26ivibGtnWr1FsuIvdM4T4HLl0d4L2WyzS1pWhuvdVbPs9g+/pV473l9TUxNqi3XETmgMI9x3r7h/jy99+lt3+IlYsX8EhNLLMXS4xd6i0XkTxR0uTYu2e66O0f4gdfq+PL2z+j3nIRKQgt7uZYYyLJ6mULeVbBLiIFpHDPoZFR5+0zXezZUsl8BbuIFJDCPYeOd/TQfWOQp7euLXQpIhJxCvccakwkmT/PeGpLZaFLEZGIU7jnUEMiSX11jNXLFhW6FBGJOIV7jlzs7efkxatakhGRoqBwz5HGRBcAe7cp3EWk8BTuOdKQSLJh9VI2r11R6FJERBTuuTAwNMJ7LZfZu22tdmsUkaKgcM+BI+eu0D80ovV2ESkaCvccaEwkWbJwHp+/f02hSxERARTu98zdOZxI8sSDFZqGJCJFQ+F+j1qS1+lM9WtJRkSKisL9HjUkkgA8/ZDCXUSKR6BwN7PnzOy0mbWY2avTHPPLZnbSzE6Y2V/ltszidTiRZNu6VazX0A0RKSIz7uduZvOB14FngU7gAzM76O4ns47ZDPw28AV3T5lZJC5je/uGaG5L8c2n7i90KSIiEwS5ct8NtLj7OXcfBN4AXpx0zDeA1909BeDuydyWWZzeOdvFyKjzjNbbRaTIBAn3DUBH1uPOzHPZtgBbzOw9MztiZs9N9YnM7GUzazKzpq6urtlVXEQaE0liyxayqypW6FJERCYIEu5T3XLpkx4vADYDe4D9wJ+a2erbfpP7AXePu3u8srK0t8UdGXXePp1kz0NrNZhDRIpOkHDvBKqyHm8ELkxxzD+4+5C7/ww4TTrsQ+t4R4pU35BaIEWkKAUJ9w+AzWa2ycwWAfuAg5OO+XvgaQAzqyC9THMul4UWm4axwRybS/t/ICISTjOGu7sPA68AbwGngDfd/YSZvWZmL2QOewu4YmYngUbgf7j7lbkquhg0JLqor4lRtmxhoUsREbnNjK2QAO5+CDg06bnvZn3swLczv0LvQk8/py5e5bef31roUkREpqQ7VGeh8XS601MtkCJSrBTus9CYSLIxtpQHNZhDRIqUwv0uDQyN8K8tl9m7VYM5RKR4Kdzv0vvnrjAwNKoWSBEpagr3u9SYSLJ04Xwe12AOESliCve74O4cPpXkCxrMISJFTuF+F84mr3O+p19dMiJS9BTud2F8MMdW3ZUqIsVN4X4XGk4l2b5uFevKNJhDRIqbwj2g3r4hmttTWpIRkZKgcA9obDCHWiBFpBQo3ANqOHWJ8uWL2FV12zb1IiJFR+EewMio8/aZLvZsqdRgDhEpCQr3AI61p+jRYA4RKSEK9wDGBnM8uUUtkCJSGhTuATQkksRrYpQt1WAOESkNCvcZnO/pJ/HpNfZu05KMiJQOhfsMGhMazCEipUfhPoPGRJKq8qU8UKnBHCJSOhTudzAwNMJ7n1xm79b7NJhDREqKwv0O3v9EgzlEpDQp3O+gITOY47FN5YUuRUTkrijcp+HuNCSSPLFZgzlEpPQo3Kdx5pIGc4hI6VK4T2N8MMdDCncRKT0K92k0JC7x2fWr+EzZkkKXIiJy1xTuU+jpG6S5TYM5RKR0Kdyn8M6ZLkYdtUCKSMlSuE+hIZFkzfJFPLxRgzlEpDQp3CcZGXXeOdPFUw9pMIeIlK5A4W5mz5nZaTNrMbNX73DcV83MzSyeuxLza2wwx96t9xW6FBGRWZsx3M1sPvA68DywHdhvZtunOG4l8N+An+S6yHw6nEiyYJ7xxS0VhS5FRGTWgly57wZa3P2cuw8CbwAvTnHc7wLfAwZyWF/eNSaSxGtjrFqiwRwiUrqChPsGoCPrcWfmuXFm9ghQ5e7/mMPa8m58MIeWZESkxAUJ96neVfTxF83mAd8HfmvGT2T2spk1mVlTV1dX8CrzZPyuVLVAikiJCxLunUBV1uONwIWsxyuBHcDbZtYKPA4cnOpNVXc/4O5xd49XVhbfsOnGRJLq8mU8ULm80KWIiNyTIOH+AbDZzDaZ2SJgH3Bw7EV373X3Cnevdfda4Ajwgrs3zUnFc6R/cIT3Wi7zzNa1GswhIiVvxnB392HgFeAt4BTwprufMLPXzOyFuS4wX94/d5mbw6PackBEQmFBkIPc/RBwaNJz353m2D33Xlb+NSSSLFs0n8fu12AOESl9ukOVzGCOU0meeLCCxQs0mENESp/CHTh96RoXege0JCMioaFwRy2QIhI+Cneg4VSSHRtWcd8qDeYQkXCIfLinbgxytD3FMxqnJyIhEvlwf/esBnOISPhEPtwPn9JgDhEJn0iH+/DIKO+c6WLPQ2uZp8EcIhIikQ73Yx099PYPsXeblmREJFwiHe6HT6UHczyxWYM5RCRcIh3ujYkkj9aWazCHiIROZMO9M9XH6UvXtCQjIqEU2XBv1F2pIhJikQ33hkSSmjXLuL9CgzlEJHwiGe79gyP8+JMrGswhIqEVyXD/8ScazCEi4RbJcB8bzLF7kwZziEg4RS7c3Z2GRJIvbtZgDhEJr8iFe+LTa1zUYA4RCbnIhfv4YA5t8SsiIRbJcP/chjLWajCHiIRYpMI9dWOQY+0p3bgkIqEXqXB/50x6MMdehbuIhFykwv1wIknFisV8bkNZoUsREZlTkQn34ZFR3jmdZM9DlRrMISKhF5lwP9rew9WBYS3JiEgkRCbcDycusXC+BnOISDREJtzHBnOs1GAOEYmASIR7R3cfZy5d112pIhIZkQj3xtPpu1IV7iISFZEI94ZEkto1y7i/ckWhSxERyYtA4W5mz5nZaTNrMbNXp3j922Z20sw+MrPDZlaT+1Jnp29wODOY475ClyIikjczhruZzQdeB54HtgP7zWz7pMOOAXF33wn8HfC9XBc6Wz9uucKgBnOISMQEuXLfDbS4+zl3HwTeAF7MPsDdG929L/PwCLAxt2XOXsPpJMs1mENEIiZIuG8AOrIed2aem85LwD9N9YKZvWxmTWbW1NXVFbzKWXJ3GhNJvri5kkULIvH2gogIECzcp7pX36c80OxrQBz4valed/cD7h5393hlZWXwKmfp1EUN5hCRaFoQ4JhOoCrr8UbgwuSDzOxLwHeAp9z9Zm7KuzdjLZB7ts79PyQiIsUkyJX7B8BmM9tkZouAfcDB7APM7BHgj4EX3D2Z+zJn5/CpS+zcWMbalRrMISLRMmO4u/sw8ArwFnAKeNPdT5jZa2b2Quaw3wNWAH9rZsfN7OA0ny5vum8McqyjR+P0RCSSgizL4O6HgEOTnvtu1sdfynFd9+ydM0ncYe82hbuIRE9oW0gOn0oP5tixXoM5RCR6QhnuwyOjvHumi6c1mENEIiqU4d7clkoP5tCSjIhEVCjDvSGRzAzmUAukiERTaMN996ZyViwO9H6xiEjohC7cO7r7OJu8rl0gRSTSQhfuDQkN5hARCWW4b6pYzqaK5YUuRUSkYEIV7n2Dw7x/7oqu2kUk8kIV7u9pMIeICBCycG9IJFmxeAGP1mowh4hEW2jC/dZgjgoN5hCRyAtNCp68eJVPrw7wtJZkRETCE+6NmRZIbfErIhKicG9IJHl4YxmVKxcXuhQRkYILRbhfuX4zPZhDSzIiIkBIwv2dM13pwRzackBEBAhJuDckklSuXMxn168qdCkiIkWh5MN9aGSUdzSYQ0RkgpIP9+a2FNcGhrULpIhIlpIP91uDOSoKXYqISNEIRbg/tmmNBnOIiGQp6XBvv9JHS/K6NgoTEZmkpMO9IXEJ0GAOEZHJSjvcT3dxf8VyajWYQ0RkgpIN9xs3hznyiQZziIhMpWTD/b2WywyOaDCHiMhUSjbcG0+nB3PENZhDROQ2JRnu7k5DIsmTWzSYQ0RkKiWZjCcuXOXS1Zvau11EZBqBwt3MnjOz02bWYmavTvH6YjP7m8zrPzGz2lwXmq0xkcQM9ijcRUSmNGO4m9l84HXgeWA7sN/Mtk867CUg5e4PAt8H/k+uC83WcDrJzo2rNZhDRGQaQa7cdwMt7n7O3QeBN4AXJx3zIvDnmY//DthrZnOyReOV6zc53tHDM7pqFxGZVpBw3wB0ZD3uzDw35THuPgz0AmtyUeBkb5/ODObYpnAXEZlOkHCf6grcZ3EMZvaymTWZWVNXV1eQ+m6zaulCnt1+nwZziIjcQZCtFDuBqqzHG4EL0xzTaWYLgDKge/IncvcDwAGAeDx+W/gH8ez2+3h2u/ZuFxG5kyBX7h8Am81sk5ktAvYBBycdcxD4z5mPvwo0uPuswltERO7djFfu7j5sZq8AbwHzgR+6+wkzew1ocveDwJ8Bf2lmLaSv2PfNZdEiInJngSZcuPsh4NCk576b9fEA8Eu5LU1ERGarJO9QFRGRO1O4i4iEkMJdRCSEFO4iIiGkcBcRCSErVDu6mXUBbbP87RXA5RyWUwp0ztGgc46GeznnGnevnOmggoX7vTCzJnePF7qOfNI5R4POORrycc5alhERCSGFu4hICJVquB8odAEFoHOOBp1zNMz5OZfkmruIiNxZqV65i4jIHRR1uBfbYO58CHDO3zazk2b2kZkdNrOaQtSZSzOdc9ZxXzUzN7OS76wIcs5m9suZ7/UJM/urfNeYawF+tqvNrNHMjmV+vr9SiDpzxcx+aGZJM/t4mtfNzP4w8+fxkZnV5bQAdy/KX6S3F/4EuB9YBHwIbJ90zK8DP8h8vA/4m0LXnYdzfhpYlvn4W1E458xxK4F3gSNAvNB15+H7vBk4BsQyj9cWuu48nPMB4FuZj7cDrYWu+x7P+UmgDvh4mte/AvwT6Ul2jwM/yeXXL+Yr96IazJ0nM56zuze6e1/m4RHSk7FKWZDvM8DvAt8DBvJZ3BwJcs7fAF539xSAuyfzXGOuBTlnB8bmZ5Zx+8S3kuLu7zLFRLosLwJ/4WlHgNVmti5XX7+Yw72oBnPnSZBzzvYS6X/5S9mM52xmjwBV7v6P+SxsDgX5Pm8BtpjZe2Z2xMyey1t1cyPIOf8O8DUz6yQ9P+I381Nawdzt3/e7EmhYR4HkbDB3CQl8Pmb2NSAOPDWnFc29O56zmc0Dvg98PV8F5UGQ7/MC0ksze0j/7+z/mdkOd++Z49rmSpBz3g/8yN1/38w+T3q62w53H5378gpiTvOrmK/c72YwN3cazF1CgpwzZvYl4DvAC+5+M0+1zZWZznklsAN428xaSa9NHizxN1WD/mz/g7sPufvPgNOkw75UBTnnl4A3Adz9fWAJ6T1YwirQ3/fZKuZwj+Jg7hnPObNE8cekg73U12FhhnN29153r3D3WnevJf0+wwvu3lSYcnMiyM/235N+8xwzqyC9THMur1XmVpBzbgf2ApjZNtLh3pXXKvPrIPCfMl0zjwO97n4xZ5+90O8oz/Bu81eAM6TfZf9O5rnXSP/lhvQ3/2+BFuDfgPsLXXMezvlfgEvA8cyvg4Wuea7PedKxb1Pi3TIBv88G/F/gJPBTYF+ha87DOW8H3iPdSXMc+HKha77H8/1r4CIwRPoq/SXgm8A3s77Hr2f+PH6a659r3aEqIhJCxbwsIyIis6RwFxEJIYW7iEgIKdxFREJI4S4iEkIKdxGREFK4i4iEkMJdRCSE/j/dKLk2OSddlwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f459b586208>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ROC curve using y_pred_class - WRONG!\n",
    "fpr, tpr, thresholds = metrics.roc_curve(y_test, y_pred_class)\n",
    "plt.plot(fpr, tpr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7809621710526315\n"
     ]
    }
   ],
   "source": [
    "# AUC using y_pred_class - WRONG!\n",
    "print(metrics.roc_auc_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you use **y_pred_class**, it will interpret the zeros and ones as predicted probabilities of 0% and 100%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cross-validation\n",
    "\n",
    "## Review of model evaluation procedures\n",
    "\n",
    "**Motivation:** Need a way to choose between machine learning models\n",
    "\n",
    "- Goal is to estimate likely performance of a model on **out-of-sample data**\n",
    "\n",
    "**Initial idea:** Train and test on the same data\n",
    "\n",
    "- But, maximizing **training accuracy** rewards overly complex models which **overfit** the training data\n",
    "\n",
    "**Alternative idea:** Train/test split\n",
    "\n",
    "- Split the dataset into two pieces, so that the model can be trained and tested on **different data**\n",
    "- **Testing accuracy** is a better estimate than training accuracy of out-of-sample performance\n",
    "- But, it provides a **high variance** estimate since changing which observations happen to be in the testing set can significantly change testing accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import metrics\n",
    "\n",
    "# define X and y\n",
    "feature_cols = ['Pclass', 'Parch', 'Age', 'Sex_Female', 'Embarked_Q', 'Embarked_S']\n",
    "X = titanic[feature_cols]\n",
    "y = titanic.Survived"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7937219730941704\n"
     ]
    }
   ],
   "source": [
    "# train/test split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)\n",
    "\n",
    "# train a logistic regression model\n",
    "logreg = LogisticRegression(C=1e9)\n",
    "logreg.fit(X_train, y_train)\n",
    "\n",
    "# make predictions for testing set\n",
    "y_pred_class = logreg.predict(X_test)\n",
    "\n",
    "# calculate testing accuracy\n",
    "print(metrics.accuracy_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7802690582959642\n"
     ]
    }
   ],
   "source": [
    "# train/test split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=2)\n",
    "\n",
    "# train a logistic regression model\n",
    "logreg = LogisticRegression(C=1e9)\n",
    "logreg.fit(X_train, y_train)\n",
    "\n",
    "# make predictions for testing set\n",
    "y_pred_class = logreg.predict(X_test)\n",
    "\n",
    "# calculate testing accuracy\n",
    "print(metrics.accuracy_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7847533632286996\n"
     ]
    }
   ],
   "source": [
    "# train/test split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=3)\n",
    "\n",
    "# train a logistic regression model\n",
    "logreg = LogisticRegression(C=1e9)\n",
    "logreg.fit(X_train, y_train)\n",
    "\n",
    "# make predictions for testing set\n",
    "y_pred_class = logreg.predict(X_test)\n",
    "\n",
    "# calculate testing accuracy\n",
    "print(metrics.accuracy_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "train test spliting create bias due to the intrinsic randomness in the sets selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# K-fold cross-validation\n",
    "\n",
    "1. Split the dataset into K **equal** partitions (or \"folds\").\n",
    "2. Use fold 1 as the **testing set** and the union of the other folds as the **training set**.\n",
    "3. Calculate **testing accuracy**.\n",
    "4. Repeat steps 2 and 3 K times, using a **different fold** as the testing set each time.\n",
    "5. Use the **average testing accuracy** as the estimate of out-of-sample accuracy.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Diagram of **5-fold cross-validation:**\n",
    "\n",
    "![5-fold cross-validation](https://raw.githubusercontent.com/justmarkham/DAT8/master/notebooks/images/cross_validation_diagram.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration                   Training set observations                   Testing set observations\n",
      "    1     [ 5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24]        [0 1 2 3 4]       \n",
      "    2     [ 0  1  2  3  4 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24]        [5 6 7 8 9]       \n",
      "    3     [ 0  1  2  3  4  5  6  7  8  9 15 16 17 18 19 20 21 22 23 24]     [10 11 12 13 14]     \n",
      "    4     [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 20 21 22 23 24]     [15 16 17 18 19]     \n",
      "    5     [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19]     [20 21 22 23 24]     \n"
     ]
    }
   ],
   "source": [
    "# simulate splitting a dataset of 25 observations into 5 folds\n",
    "from sklearn.cross_validation import KFold\n",
    "kf = KFold(25, n_folds=5, shuffle=False)\n",
    "\n",
    "# print the contents of each training and testing set\n",
    "print('{} {:^61} {}'.format('Iteration', 'Training set observations', 'Testing set observations'))\n",
    "for iteration, data in enumerate(kf, start=1):\n",
    "    print('{:^9} {} {:^25}'.format(str(iteration), str(data[0]), str(data[1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Dataset contains **25 observations** (numbered 0 through 24)\n",
    "- 5-fold cross-validation, thus it runs for **5 iterations**\n",
    "- For each iteration, every observation is either in the training set or the testing set, **but not both**\n",
    "- Every observation is in the testing set **exactly once**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create k-folds\n",
    "kf = KFold(X.shape[0], n_folds=10, random_state=0)\n",
    "\n",
    "results = []\n",
    "\n",
    "for train_index, test_index in kf:\n",
    "    X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n",
    "    y_train, y_test = y.iloc[train_index], y.iloc[test_index]\n",
    "\n",
    "    # train a logistic regression model\n",
    "    logreg = LogisticRegression(C=1e9)\n",
    "    logreg.fit(X_train, y_train)\n",
    "\n",
    "    # make predictions for testing set\n",
    "    y_pred_class = logreg.predict(X_test)\n",
    "\n",
    "    # calculate testing accuracy\n",
    "    results.append(metrics.accuracy_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    10.000000\n",
       "mean      0.794644\n",
       "std       0.030631\n",
       "min       0.764045\n",
       "25%       0.768820\n",
       "50%       0.780899\n",
       "75%       0.820225\n",
       "max       0.842697\n",
       "dtype: float64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(results).describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import cross_val_score\n",
    "\n",
    "logreg = LogisticRegression(C=1e9)\n",
    "\n",
    "results = cross_val_score(logreg, X, y, cv=10, scoring='accuracy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    10.000000\n",
       "mean      0.794665\n",
       "std       0.019263\n",
       "min       0.775281\n",
       "25%       0.779963\n",
       "50%       0.786517\n",
       "75%       0.806742\n",
       "max       0.829545\n",
       "dtype: float64"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(results).describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparing cross-validation to train/test split\n",
    "\n",
    "Advantages of **cross-validation:**\n",
    "\n",
    "- More accurate estimate of out-of-sample accuracy\n",
    "- More \"efficient\" use of data (every observation is used for both training and testing)\n",
    "\n",
    "Advantages of **train/test split:**\n",
    "\n",
    "- Runs K times faster than K-fold cross-validation\n",
    "- Simpler to examine the detailed results of the testing process\n",
    "\n",
    "## Cross-validation recommendations\n",
    "\n",
    "1. K can be any number, but **K=10** is generally recommended\n",
    "2. For classification problems, **stratified sampling** is recommended for creating the folds\n",
    "    - Each response class should be represented with equal proportions in each of the K folds\n",
    "    - scikit-learn's `cross_val_score` function does this by default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Improvements to cross-validation\n",
    "\n",
    "**Repeated cross-validation**\n",
    "\n",
    "- Repeat cross-validation multiple times (with **different random splits** of the data) and average the results\n",
    "- More reliable estimate of out-of-sample performance by **reducing the variance** associated with a single trial of cross-validation\n",
    "\n",
    "**Creating a hold-out set**\n",
    "\n",
    "- \"Hold out\" a portion of the data **before** beginning the model building process\n",
    "- Locate the best model using cross-validation on the remaining data, and test it **using the hold-out set**\n",
    "- More reliable estimate of out-of-sample performance since hold-out set is **truly out-of-sample**\n",
    "\n",
    "**Feature engineering and selection within cross-validation iterations**\n",
    "\n",
    "- Normally, feature engineering and selection occurs **before** cross-validation\n",
    "- Instead, perform all feature engineering and selection **within each cross-validation iteration**\n",
    "- More reliable estimate of out-of-sample performance since it **better mimics** the application of the model to out-of-sample data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overfitting, Underfitting and Model Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've gone over the basics of validation, and cross-validation, it's time to go into even more depth regarding model selection.\n",
    "\n",
    "The issues associated with validation and \n",
    "cross-validation are some of the most important\n",
    "aspects of the practice of machine learning.  Selecting the optimal model\n",
    "for your data is vital, and is a piece of the problem that is not often\n",
    "appreciated by machine learning practitioners.\n",
    "\n",
    "Of core importance is the following question:\n",
    "\n",
    "**If our estimator is underperforming, how should we move forward?**\n",
    "\n",
    "- Use simpler or more complicated model?\n",
    "- Add more features to each observed data point?\n",
    "- Add more training samples?\n",
    "\n",
    "The answer is often counter-intuitive.  In particular, **Sometimes using a\n",
    "more complicated model will give _worse_ results.**  Also, **Sometimes adding\n",
    "training data will not improve your results.**  The ability to determine\n",
    "what steps will improve your model is what separates the successful machine\n",
    "learning practitioners from the unsuccessful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Illustration of the Bias-Variance Tradeoff\n",
    "\n",
    "For this section, we'll work with a simple 1D regression problem.  This will help us to\n",
    "easily visualize the data and the model, and the results generalize easily to  higher-dimensional\n",
    "datasets.  We'll explore a simple **linear regression** problem.\n",
    "This can be accomplished within scikit-learn with the `sklearn.linear_model` module.\n",
    "\n",
    "We'll create a simple nonlinear function that we'd like to fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "def test_func(x, err=0.5):\n",
    "    y = 10 - 1. / (x + 0.1)\n",
    "    if err > 0:\n",
    "        y = np.random.normal(y, err)\n",
    "    return y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create a realization of this dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_data(N=40, error=1.0, random_seed=1):\n",
    "    # randomly sample the data\n",
    "    np.random.seed(1)\n",
    "    X = np.random.random(N)[:, np.newaxis]\n",
    "    y = test_func(X.ravel(), error)\n",
    "    \n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAEw1JREFUeJzt3X+MHOddx/HPt2cHLiXlUnyF+hLnHJQchETC0apNsVTapsUhlNiKIkikQFoFrBZRoCAjm/6RCoRiYWgpUgRYpTSF0qSk1tVqAqbEiQJRE3rOhTo/ejSk+eG1aa4KFxBcydX58sfu2ef1zu7szuzMM8+8X1KUvb3JzfPk9j777Pd55hlzdwEAqu91ZTcAAJAPAh0AIkGgA0AkCHQAiASBDgCRINABIBIEOgBEgkAHgEgQ6AAQiXVFnmzDhg0+PT1d5CkBoPKOHDnybXef7HdcoYE+PT2tubm5Ik8JAJVnZs+nOY6SCwBEgkAHgEgQ6AAQCQIdACJBoANAJApd5QIgLLPzTe07tKDjS8vaODGuXdtmtGPLVNnNwpAIdKCmZueb2nPgqJZXTkqSmkvL2nPgqCQR6hVFyQWoqX2HFk6F+arllZPad2ihpBYhKwIdqKnjS8sDPY/wEehATW2cGB/oeYSPQAdqate2GY2vHzvjufH1Y9q1baakFiErJkWBmlqd+GSVSzwIdKDGdmyZIsAjQskFACJBoANAJAh0AIgEgQ4AkWBSFEBlsPdMbwQ6gEpg75n+KLkAqAT2numvb6Cb2afM7CUze2LNc280sy+b2Tfa/z5/tM0EUHfsPdNfmhH6pyVd0/Hcbkn3u/slku5vfw0AI8PeM/31DXR3f0jSyx1Pb5d0Z/vxnZJ25NwuADhDyHvPzM43tXXvYW3efa+27j2s2flmKe0YdlL0B939hCS5+wkze1OObQKAs4S690xIk7UjX+ViZjsl7ZSkTZs2jfp0ACIW4t4zvSZri27rsKtcvmVmb5ak9r9fSjrQ3fe7e8PdG5OTk0OeDgDCFNJk7bCBflDSLe3Ht0j6Yj7NAYBqCWmyNs2yxc9J+oqkGTM7Zma3Stor6T1m9g1J72l/DQC1E9Jkbd8aurvflPCtq3NuCwAUJq9tBEKarOXSfwC1k/fKlFAma7n0H0DtxLqNACN0oAbYpfBMIa1MyROBDgypKiEZ0oUvodg4Ma5ml/Cu+jYClFyAIayGZHNpWa7TIVnWJd+9xFpeyCKklSl5ItCBIVQpJGMtL2SxY8uUbr/+Ck1NjMskTU2M6/brr6j8JxZKLsAQqhSSsZYXBpFUHqt6gHdihA4MIaSrA/uJtbyQVpXKY1kR6MAQqhSSsZYX0vrowScrUx7LipILMISQrg5MI6m8UJWVOsOanW9qaXml6/dCLI9lRaCjMkILn6rXYOuwnLHXKDzE8lhWBDoqoQ7hU7SQ9vEe1Zt1r1F4iOWxrKihoxKqtEywKkJZqTPKScukUfj5566PciBAoKMSQgmfmISyUmeUb9ZJk9e3/eyPZf7ZISLQUQmhhE9MQlmpM8o367qt8KGGjkrYtW3mjBq6FO4ywSpYrVkvr5zUmJlOumuqpInmUV/4VPXJ60EwQkclrB1pSdKY2amP5TFeIDJKa2vWknTS/dSbYxnBV8Qnhdn5prbuPazNu+/V1r2Ho33NEOiojB1bpk798Z90lxT3VX+jEtoE86jLInW6UpSSCyolpKV2VTWKmnXWZYejLIvU6TVDoGMgZV/cw2qX7PKuWYd+jUCdXjOUXJBaCB9dWe2SXd4169BKOJ3KeM2UVbMn0JFaCH+4oSy1q7K8a9ahj4CLfs2UOfCh5BK4sksca4Xwh1u1TbFClWfNOvT91ot+zZRZsyfQAxZabTKUP9w6rSuugipcI1Dka6bMgQ8ll4CFUOJYi3IHuqnb1Zj9lDnPwwg9YCGUONai3IEkfGo6rcxPLAR6wEIpcazFH279hDSPUwVlDnwyBbqZfVjSL0lySUclvd/dv5NHw1CN2iTiFto8TlWUNfAZuoZuZlOSfk1Sw90vlzQm6ca8GgZqkxitNGulQ5vHQW9ZSy7rJI2b2YqkcyUdz94krEWJA6OQduQd2jwOeht6hO7uTUl/KOkFSSckveLu/5BXw4Bu6rJr3qilHXlzZW61ZCm5nC9pu6TNkjZKer2Z3dzluJ1mNmdmc4uLi8O3FLUXwtYDsUg78maparVkWYf+bknfdPdFd1+RdEDST3Qe5O773b3h7o3JyckMp0MIyhwhU8/NT9qRN/M41ZKlhv6CpKvM7FxJy5KuljSXS6sQpLJXPFDPzc8gK6iYx6mOLDX0RyXdI+kxtZYsvk7S/pzahQCVPUKmnpsfRt5xyrTKxd1vk3RbTm1B4MoeIYe6Lj/pwpvQL8hh5B0frhRFamVfuRri1gNJZai551/WF440uSAHhTJv35uxCI1Gw+fmKLNXVWd4Sa0Rcp0/qm/de7jrm9yY2an7nq41NTGuh3e/q4imISJmdsTdG/2OY4SO1EIYIYdWxkgqN3UL817HA3kg0DGQNHXXUYVu2atsukkqQyWN0JnAxSixHzpylcfFP0lr3cteZdNN0oU3N731Qi7IQeEYoSNXWW+/1WsUXvYqm256laEaF70xqPIQ4kegI1dZQ7fXG0LZq2ySJJWhWBaIolFyQa6yXvzT6w2BfUWA3gh05Cpr6PZ6Q+DqRqA3Si7IVdaljf2uBqWMASQj0JG7LKEbwlp3oKoIdASHUTgwHGroABAJAh0AIkGgA0AkCHQAiASBDgCRYJULegptu1oAyQh0JApxu1oAySi5IFGI29UCSEagI1GI29UCSEagI1HWnRMBFItARyK2qwWqhUlRJGKjLKBaCHT0xEZZQHVQcgGASBDoABCJTIFuZhNmdo+Zfd3Mnjazt+XVMADAYLLW0D8h6e/d/QYzO0fSuTm0CQAwhKED3czeIOntkt4nSe7+qqRX82kWAGBQWUouF0talPSXZjZvZp80s9fn1C4AwICyBPo6SVdK+lN33yLpfyTt7jzIzHaa2ZyZzS0uLmY4XT3Mzje1de9hbd59r7buPazZ+WbZTQJQEVkC/ZikY+7+aPvre9QK+DO4+353b7h7Y3JyMsPp4re6u2FzaVmu07sbEuoA0hg60N39PyS9aGar14FfLempXFoVkUFG3OxuCCCLrKtcPiTps+0VLs9Ken/2JsVj0P3E2d0QQBaZAt3dH5fUyKkt0ek14u4W6BsnxtXsEt4bJ8a5cxCAvrhSdIQGHXEn7W74zh+ZpLYOoC8CfYQG3U98x5Yp3X79FZqaGJdJmpoY1+3XX6EHvr5IbR1AX+y2OEK7ts2cUUOX+u8n3m13ww/f/XjXY6mtA1iLEfoIJY24B619c+cgAGkwQu8hj4nIPPYTH2akD6B+CPQEgy45HCXuHAQgDQI9waBLDkeNOwcB6IdAT1DERT6sLQeQJwI9Qa+LfNLoF9YhlXQAxIFVLgmSLvJJMxGZZpMt9m0BkDcCPUGWJYdpwpp9WwDkjZJLF53lko///I8PVAZJE9ZZSzoA0IkReoc89iRPcyFQlpIOAHRDoHfIo7adJqzzuooUAFZRcumQVC5pLi1r8+57Uy0vTHshEGvLAeSJQO+QVNuWdEYJRuq9vJCwBlA0Si4dupVLOrG8EECIGKF36CyXeMJxLC8EEBoCvYu15ZKtew+zvBBAJVBy6YPlhQCqghF6H2xdC6AqCPQUWLECoAoouQBAJAh0AIgEgQ4AkSDQASASBDoARCJzoJvZmJnNm9mX8mgQAGA4eYzQf13S0zn8HABABpkC3cwukPQzkj6ZT3MAAMPKOkL/Y0m/Lem1HNoCAMhg6CtFzey9kl5y9yNm9o4ex+2UtFOSNm3aNOzpUum8FyiX6AOokywj9K2SrjOz5yTdJeldZvbXnQe5+353b7h7Y3JyMsPpesvjXqAAUGVDB7q773H3C9x9WtKNkg67+825tWxAedwLFACqLJp16Ek3nOBGFADqIpdAd/cH3f29efysYSXdcIIbUQCoi2hG6NyIAkDdRbMfOjeiAFB30QS6xI0oANRbNCUXAKg7Ah0AIkGgA0AkCHQAiASBDgCRINABIBIEOgBEgkAHgEgQ6AAQCQIdACJBoANAJAh0AIgEgQ4AkSDQASASBDoARIJAB4BIRHWDi06z803uYASgNqIN9Nn5pvYcOKrllZOSpObSsvYcOCpJhDqAKEVbctl3aOFUmK9aXjmpfYcWSmoRAIxWtIF+fGl5oOcBoOqiDfSNE+MDPQ8AVRdtoO/aNqPx9WNnPDe+fky7ts2U1CIAGK1oJ0VXJz5Z5QKgLqINdKkV6gQ4gLoYOtDN7EJJn5H0Q5Jek7Tf3T+RV8PSYq05ALRkGaF/V9JvuftjZnaepCNm9mV3fyqntvXFWnMAOG3oSVF3P+Huj7Uf/7ekpyUVmqKsNQeA03JZ5WJm05K2SHo0j5+XFmvNAeC0zIFuZt8n6QuSfsPd/6vL93ea2ZyZzS0uLmY93RlYaw4Ap2UKdDNbr1aYf9bdD3Q7xt33u3vD3RuTk5NZTncW1poDwGlZVrmYpL+Q9LS7fyy/JqXHWnMAOC3LKpetkn5B0lEze7z93O+4+33Zm5Uea80BoGXoQHf3f5ZkObYFAJBBtHu5AEDdEOgAEAkCHQAiQaADQCQIdACIBIEOAJEg0AEgEgQ6AESCQAeASBDoABAJAh0AIkGgA0AkCHQAiASBDgCRINABIBIEOgBEgkAHgEgQ6AAQCQIdACJBoANAJAh0AIgEgQ4AkSDQASASBDoARIJAB4BIEOgAEAkCHQAikSnQzewaM1sws2fMbHdejQIADG7oQDezMUl3SPppSZdJusnMLsurYQCAwWQZob9F0jPu/qy7vyrpLknb82kWAGBQ6zL8t1OSXlzz9TFJb83WnN5m55vad2hBx5eWtXFiXLu2zWjHlqlRnhIAKiNLoFuX5/ysg8x2StopSZs2bRr6ZLPzTe05cFTLKyclSc2lZe05cFSSCHUAULaSyzFJF675+gJJxzsPcvf97t5w98bk5OTQJ9t3aOFUmK9aXjmpfYcWhv6ZABCTLIH+VUmXmNlmMztH0o2SDubTrLMdX1oe6HkAqJuhA93dvyvpVyUdkvS0pM+7+5N5NazTxonxgZ4HgLrJtA7d3e9z90vd/Yfd/ffzalQ3u7bNaHz92BnPja8f065tM6M8LQBURpZJ0UKtTnyyygUAuqtMoEutUCfAAaA79nIBgEgQ6AAQCQIdACJBoANAJAh0AIiEuZ+1/croTma2KOn5HH7UBknfzuHnVE0d+13HPkv0u07S9Pkid++7d0qhgZ4XM5tz90bZ7ShaHftdxz5L9LvsdhQpzz5TcgGASBDoABCJqgb6/rIbUJI69ruOfZbod53k1udK1tABAGer6ggdANAh6EA3s2vMbMHMnjGz3V2+/z1mdnf7+4+a2XTxrcxXij7/ppk9ZWZfM7P7zeyiMtqZt379XnPcDWbmZhbFSog0/Tazn2v/zp80s78puo15S/Ea32RmD5jZfPt1fm0Z7cybmX3KzF4ysycSvm9m9ift/y9fM7MrBz6Juwf5j6QxSf8u6WJJ50j6V0mXdRzzK5L+rP34Rkl3l93uAvr8Tknnth9/sOp9Ttvv9nHnSXpI0iOSGmW3u6Df9yWS5iWd3/76TWW3u4A+75f0wfbjyyQ9V3a7c+r72yVdKemJhO9fK+nv1Lpf81WSHh30HCGP0N8i6Rl3f9bdX5V0l6TtHcdsl3Rn+/E9kq42s243r66Kvn129wfc/X/bXz6i1r1cqy7N71qSfk/SH0j6TpGNG6E0/f5lSXe4+39Kkru/VHAb85amzy7pDe3H368u9yquInd/SNLLPQ7ZLukz3vKIpAkze/Mg5wg50Kckvbjm62Pt57oe461b4r0i6QcKad1opOnzWreq9Y5edX37bWZbJF3o7l8qsmEjlub3famkS83sYTN7xMyuKax1o5Gmzx+VdLOZHZN0n6QPFdO00g3693+WkG9w0W2k3bkkJ80xVZK6P2Z2s6SGpJ8caYuK0bPfZvY6SR+X9L6iGlSQNL/vdWqVXd6h1qexfzKzy919acRtG5U0fb5J0qfd/Y/M7G2S/qrd59dG37xSZc6zkEfoxyRduObrC3T2R69Tx5jZOrU+nvX6SBO6NH2Wmb1b0kckXefu/1dQ20apX7/Pk3S5pAfN7Dm16osHI5gYTfsa/6K7r7j7NyUtqBXwVZWmz7dK+rwkuftXJH2vWvudxC7V338vIQf6VyVdYmabzewctSY9D3Ycc1DSLe3HN0g67O3ZhYrq2+d26eHP1QrzqtdTV/Xst7u/4u4b3H3a3afVmju4zt3nymlubtK8xmfVmgiXmW1QqwTzbKGtzFeaPr8g6WpJMrMfVSvQFwttZTkOSvrF9mqXqyS94u4nBvoJZc/89pkVvlbSv6k1K/6R9nO/q9Yfs9T6Rf+tpGck/Yuki8tucwF9/kdJ35L0ePufg2W3uYh+dxz7oCJY5ZLy922SPibpKUlHJd1YdpsL6PNlkh5WawXM45J+quw259Tvz0k6IWlFrdH4rZI+IOkDa37Xd7T/vxwd5jXOlaIAEImQSy4AgAEQ6AAQCQIdACJBoANAJAh0AIgEgQ4AkSDQASASBDoAROL/AZsGnJzp4mhRAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f45807ceda0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X, y = make_data(40, error=1)\n",
    "plt.scatter(X.ravel(), y);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now say we want to perform a regression on this data.  Let's use the built-in linear regression function to compute a fit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl8VNX9//HXxwASEAUFlVUEhZ9iVWz6xaoVVJSiFahb3cWq1A1pa7Fu/Wq/aqWiVVRccEPApS4UEVFKpa1KixYEEaEogiwBEYUgSIAQzu+Pc6NDyCSz35mb9/PxyIOZO5O5n5uE95w595xzzTmHiIgUvl3CLkBERDJDgS4iEhEKdBGRiFCgi4hEhAJdRCQiFOgiIhGhQBdJgJk5Mzsg7DpEaqNAF4kgM2tkZi+Z2WfBm1GvOp6/sdpXpZk9EPP4pWa2KHjsDTNrk/WDkKQp0KXeMbOiEPfdIJFtyb5GHO8A5wOf1/VE59xuVV/APkA58GKwv57AH4D+wJ7AEuC5ZGqW3FCgR1TQMhtqZnPN7Bsze8LM9jGz181sg5n9zcxaxDz/SDP7l5mVmdkHsS06M7vYzBYE37fYzH4R81gvM1thZtea2RdmtsrMLq6lroHBa2wwsyVmdl6wvcjM7jazL4PHrwpalg1ijqd3zOvcambjYu6/aGafm9l6M3vLzLrFPDbazB42s8lm9g1wnJntGuxvmZmtNrNHzKw45nuGBsey0sx+XsfPeo/g57vKzErN7PaqN43geKeb2b1mtha4Nc62XczsZjNbGvwcx5jZHsFrdAx+FpeY2TJgWm31ADjntjrn7nPOvQNU1vX8as4AvgDeDu6fCrzonPvIObcVuA041sw6J/m6kmUK9Gg7HTgR6IL/T/k6cCPQEv+7vwbAzNoCrwG341tgvwFeNrNWwet8AfwE2B24GLjXzI6I2c++wB5AW+ASYGTsm0UVM2sK3A/0dc41A44C5gQPXxbsoztQgg+VZLwOHAjsDbwPPFPt8XOBO4Bm+JbrH/E/l8OBA4La/zeo88fBz+DE4DV7U7ungW3B63QHTgIujXm8B7A4qO2OONsGBl/HAZ2A3YAHq+2nJ3AQ0Ceoc66ZnVtHbam4CBjjvlsXxIIvYu4DHJKFfUs6nHP6iuAX8BlwXsz9l4GHY+4PBiYEt38LjK32/VOAi+K89gRgSHC7F/7jeYOYx78Ajqzh+5oCZfg3muJqj00DLo+5fxLgql43OJ7eMY/fCoyLU1/z4Hv3CO6PxgdU1eMGfAN0jtn2Q2BJcPtJYFjMY12C1zughn3tA2yJPR7gHODvwe2BwLJq31PTtjeBK2PudwUqgAZAx2D/nVL8W1gB9ErwuR3wLfr9Y7adAHwJHAoUA48C24Fzwv4719eOX2qhR9vqmNvlNdzfLbi9H3Bm0N1SZmZlwDFAawAz62tmM8xsbfDYyfhWfpWvnHPbYu5vinntbznnvgF+BlwOrDKz18zs/wUPtwGWxzx9aaIHGXTXDDOzT83sa3z4U63G2NduBTQBZsUc7xvB9mRr2Q9oGBxP1Ws9im9517TveNvaVNvPUnyY71PH62TahcA7zrklVRucc28Ct+AbBUvxP98N+DcKySMKdAEfFGOdc81jvpo654aZ2a74/8h3A/s455oDk9nxI3jCnHNTnHMn4t8s/gs8Fjy0Cmgf89QO1b71G3wIV9k35va5+BN2vfFdPx2D7bE1xi4r+iX+Da1bzPHu4fwJwURqibUc30JvGfNauzvnusU8p6YlTatvW4l/c4jd5zZ2fBPOxdKoF+K7kHbgnBvpnDvQObc3/u+hATAvB/VIEhToAjAOONXM+gSt3cbByc52QCNgV2ANsM3M+uK7Q5IWnJTtF/SlbwE28t0JuxeAa8ysXdD/fn21b58DnG1mDc2seh97s+D1vsKH/h9qq8M5tx3/RnKvme0d1NbWzPrE1DLQzA42syb41mm811oF/BW4x8x2D05udjY/MiQZzwG/MrP9zWy34Bj+XO2TT1KCE7+Ng7uNgt9r3DdiMzsKfy7hxWrbG5vZIeZ1AEYBI5xz61KtTbJDgS4455bjW7g34oN7OTAU2MU5twF/8vQFYB2+NTwxxV3tAlyLb42uxZ/kuzJ47DF8v/0H+JOa46t97++AzkENvweejXlsDL4roBSYD8xIoJbfAouAGUE3zd/w/dY4514H7sP36y+i7lElF+Lf+OYH9b1E0F2VhCeBscBb+GGBm/HnOeIys4+qRgnFsRD/SaQt/mdbTvApwMxuNLPXqz3/ImB88DuP1Rj/894IvAf8G//7kDxjzukCF5J/zKwjPtgaptNKFalP1EIXEYkIBbqISESoy0VEJCLUQhcRiYikFgVKV8uWLV3Hjh1zuUsRkYI3a9asL51zrep6Xk4DvWPHjsycOTOXuxQRKXhmltDMaXW5iIhEhAJdRCQiFOgiIhGhQBcRiQgFuohIRCjQRUQiQoEuIhIRCnQRkWz64AP41a+goiLru1Kgi4hkWmUlTJgAvXrB4YfDqFHw4YdZ360CXUQkU8rK4E9/ggMPhJ/+FJYsgeHDYcUKOOKIrO8+p1P/RUQi6eOP4f77YfRo+OYb+NGPfJD37w8NchezCnQRkVQ4B1OnwogRMHkyNGoE55wDQ4ZA9+6hlKRAFxFJxjffwNixvkW+YAHssw/ceitcfrm/HSIFuohIIpYtg5Ej4bHHYN06+P73YcwYOOss2HXXsKsDFOgiBW/C7FKGT1nIyrJy2jQvZmifrgzo3jbssqLBOZg+3Xer/OUv/v5pp8EvfwlHHQVmYVe4AwW6SAGbMLuUG8Z/SHlFJQClZeXcMN4Pj1Oop2HLFnjhBbjvPnj/fWjRAq69Fq66Cjp0CLu6uDRsUaSADZ+y8Nswr1JeUcnwKQtDqqjArV4Nv/897LcfXHghlJfDI4/A8uXwxz/mdZiDWugiBW1lWXlS2yWO99/33SrPPw9bt8LJJ/vRKieemHfdKrVRoIsUsDbNiymtIbzbNC8OoZoCs20bvPKKD/K334amTWHQIBg8GLp0Cbu6lKjLRaSADe3TleKGRTtsK25YxNA+XUOqqACsWwd33w2dO8MZZ/julHvu8bM5H3igYMMc1EIXKWhVJz41yiUB//2vHzv+9NOwaRP07Olb56eeCkVFdX9/AVCgixS4Ad3bKsDj2b4dpkzxwT1lih8vfu65cM01ftGsiFGgi0j0bNzoJ/3cfz8sXAitW8Ntt8EvfgGtWoVdXdYo0EUkOj77DB58EB5/HNavhx/8AMaNgzPP9GutRJwCXUQKm3N+lMqIEX4NcjN/snPIEDjyyIIadpguBbqIFKbNm/248REjYM4c2HNPuO46uPJKaN8+7OpCoUAXkYzK+toyq1bBww/7GZxr1kC3bv6KQOedB02aZG4/BUiBLiIZk9W1ZWbO9K3xP//ZTwo65RS/SNbxx9erbpXaaGKRiGRMxteW2bbNL5J19NH+BOeECXDFFf4KQa++CiecoDCPUWegm9mTZvaFmc2L2banmU01s0+Cf1tkt0wRKQQZW1tm7Vq/GFanTvCzn8Hnn/uVD0tLfSv9gAMyUG30JNJCHw38uNq264E3nXMHAm8G90Wknou3hkzCa8t89JEfK96uHVx/vb/Y8iuv+Bb5kCGw++4ZrDZ66gx059xbwNpqm/sDTwe3nwYGZLguESlAKa0ts307vPYanHQSHHKInxB03nkwdy68+Sb065fw1PwJs0s5etg09r/+NY4eNo0Js0vTOZyCk+pJ0X2cc6sAnHOrzGzveE80s0HAIIAOeb6WsIikJ6m1ZTZsgNGj/YJYn3wCbdrAHXf4FQ9btkx637rYB5hzru4nmXUEJjnnDgnulznnmsc8vs45V2c/eklJiZs5c2bq1YpI4Vu82M/mfOIJ+PprP/lnyBA4/XRo2DDllz162LQalxJu27yY6dcfn07FoTOzWc65krqel2oLfbWZtQ5a562BL1J8HRGpD5yDf/zDn9CcONF3oZx5pg/yHj0ysgtd7CP1QJ8IXAQMC/59JWMViUh0lJfDs8/6RbLmzvVdKTfe6Icett25GySdSUm62EdiwxafA/4NdDWzFWZ2CT7ITzSzT4ATg/siIt7KlXDzzf4anJde6lvoTzwBy5bB7bfHDfMbxn9IaVk5ju/6wBM9samLfSTQQnfOnRPnoRMyXIuIFLr33vPjxV98ESor/QiVIUOgV686JwDVNikpkVa6Lvahqf8iBSPra6SkqqICXn7Z94/PmOHHig8eDFdf7ScGJSgTfeD1/WIfCnSRApCXQ/K+/NIvivXQQ34G5wEH+L7ygQOhWbOkX0594OnTWi4idciHySoZXyMlHfPmwWWX+SVqb7oJDj4YJk3yVwYaPDilMAf1gWeCWugitciXlnHoQ/IqK/1szhEjYNo0KC6GCy/01+bs1i0ju1AfePoU6CK1SPdEXaaE1h3x9dfw1FO+K2XxYr/GyrBhfuTKXnul/fI1nRco9ElAYVKXi0gtQm8ZB3LeHbFokR+d0q6dX3N83339MrZLlsBvf5uxME9nmKLsTIEuUou0Vw/MkAHd23Lnad+jbfNiDD+d/c7TvpfZTwnO+cWwTj0VunTxVwXq3x/+8x+YPt3P7GyQuQ/1eXVeICLU5SJSi6F9uu7Qhw7hnajL2pC88nIYN853q8ybB61awe9+B5dfDq1bZ35/gZq6kGrbLnVToEvBCGMcdiGcqEv557JiBYwc6Ycerl0Lhx/u+8vPPhsaN8563UVmVNawOGCRrkCUMgW6FIQwR5vk82SVpH8uzvnJPyNGwEsv+fv9+/t+8h/9KKHLuWXqjbWmMK9tu9RNfehSENTfWrOEfy5bt/pFsnr0gKOOgjfe8CH+6acwfjwce2zCYZ6pE5lt45yHiLdd6qZAl4KQL6NN8k2dP5c1a/xiWB07+qsArV/vu1lWrIC77/bbk5DJN1ZNJMo8dblIQdC08JrF+7kcW74Sfv5z3yrfsgX69PGrHfbpA7uk3o7L5BtrIZyfKDQKdCkI+TTaJF9MmF3KN1u2fXt/l+2V9F70Hpe8/yo9ls6FJk18qA8eDAcdlJF9ZvqNNZ/PTxQiBboUhKr/9LdO/Iiy8goAGjesvz2GsSdDm235hrM++CsXvT+JDutXs2nftnDXXX42Z4s6rwyZlEy/sebtCpIFSoEuBWXLtu3f3l63qSL8FQdDMnzKQvZdvYyLZr3KmR/+jaYVm3m3XTce/cnl3DH65oxOAIqVyW6SfFknJ0oU6FIw8mVdlVA5B1OncttjN3H84plsKWrApIOO5cnv9+OjfQ/AgDuSCPNUWsiZ6ibR7zPzFOiStlx9bK7XI102bYKxY/348QULOGy3Ftx79Lk8070vXzb9rlslmb7ssFvI9fr3mSUKdElLLkOhXo50WbbMDzN87DFYtw6OOALGjGF6l6MZNenjtPqyw24hZ/v3WR/75+vvWaWIyvXFGHI54afejFt2Dt55xy+G1amTHy9+wgnw9tswcyZccAH9enRKe7GusFvI2fx91teVHNVCj5AwPkLnMhQiP255yxa/RO2IETBrFjRvDr/+NVx1Fey3305PT7cvO+xPPNn8fYb96SMsCvQICeOPONehEMlxy6tXwyOP+OVqV6/2Y8YffhguuACaNs3abvNhbH+2fp9hf/oIi7pcIiSMP+J60w2SDbNn+wsqd+gAt94K3/8+TJkCH33kl67NYphDjtZYD0m+rGOfa2qhR0gYH6Ej3w2Sadu2wSuv+G6Vt9/2oX3ZZX42Z9fMvwnWdWIwkp94yI9PH2FQoEdIWH/EUQ2FjFq3zq+l8uCDsHSpXxTrnnv81PzmzbOyy7CHJYapvjY00gp0M/sVcCnggA+Bi51zmzNRmCSvvv4R57X//tdfCejpp/1Y8p494d57oV8/KCqq+/vTUF9PDFapjw2NlAPdzNoC1wAHO+fKzewF4GxgdIZqkxTUxz/ivLN9u+8LHzHC/7vrrnDuuXDNNf6qQBlUW5dKfT0xWJ+l2+XSACg2swqgCbAy/ZJEapb3E0U2boQxY3yLfOFC2HdfuO02GDQI9t4747urq0sl7GGJknspj3JxzpUCdwPLgFXAeufcX6s/z8wGmdlMM5u5Zs2a1CuVei2vJ4p89hn85jfQrp0fM7777v6iy0uXws03ZyXMoe5JXRqBVP+kHOhm1gLoD+wPtAGamtn51Z/nnBvlnCtxzpW0atUq9Uolb+VidmreXYLOOXjrLTj9dOjcGe67z1884l//gnff9VcHatQoqyXU1aUS5WGJUrN0ulx6A0ucc2sAzGw8cBQwLhOFSWHI1UiKvOkP3rwZnn/e94/PmQN77gnXXQdXXgnt2+e0lES6VHROpX5JZ2LRMuBIM2tiZgacACzITFlSKHLVcg59osjnn8Mtt/gp+BdfDBUVMGoULF8Od96Z8zAHdanIzlJuoTvn3jWzl4D3gW3AbGBUpgqTwpCrlnNoE0VmzoQRI9j+/J+hcht/71TCxFN/w3FXngNmDL9/RmgnaTVMVapLa5SLc+4W4JYM1SIFKFcjKXIaXtu2wV/+4rtVpk+noklTnut+Mk90P4WlLdoAMPmluWBQUemA8CbtqEtFYmmmqKQlly3nrIfX2rV+3fGRI31XSqdOcO+9nFLWiY8379i1UbHd7fTt9WnSjuQnBbqkJdcf+7MyFn3+fN8aHzsWysvh+OP9FP1TToGiIj65/rWEX0qTdiRMCnRJW10t50yFcEZH1GzfDq+/7oN86lRo3BjOP9/P5vze93Z4arxupZpo0o6ESYEuWZVqCNf0JpCRtUk2bIDRo+GBB+CTT6BNG7jjDj+bs2XLGr+lpm6lhrvYDn3ooBEmEj4FumRVKiEc702g+utUSaibY/Fi343yxBPw9dfQowc895yfGNSwYa3fGq9bqaZt6j+XMCnQJatSGdYY702gyIxKt/PJyLjdHM7BP//pZ3FOnOhXNzzzTBgyxAd6EuJ1KynAJZ8o0CWrUhnWGC/sK52juGFR3SNqNm+GZ5/1/eNz58Jee8GNN8IVV0BbBbBEly5BJ1mVymzGeGFftRZJ3LVJVq70i2G1bw+XXOJb6I8/7ocg3n67wlwiTy10yapUhjXWNra9xq6P997zrfEXXoDKSn/xiCFDoFcvMMvGYYnkJQW6ZF2yE4ISehOoqICXX/ZBPmOGX7L26qv9V+fOmT4EkYKgQJe8FPdN4Kuv/KJYI0dCaSkccIC/oMTAgdCsWc7rFMknCnQpDPPm+db4uHH+pGfv3vDoo9C3L+yiU0EioECXfLZ9O7z2mh92OG0aFBfDhRf62ZzduoVdnUjeUaBL/vn6a3jqKT+b89NP/aXdhg2DSy/1QxBFpEYKdMkfixb5EH/qKT9F/6ij4A9/gJ/+tM7ZnCKiQJewOee7U0aMgEmToEEDOOssP+zwBz8IuzqRgqJAl5SkvYJiebk/wXn//f6EZ6tWflLQFVdA69bZK1wkwhTokrS0lrFdsQIeesiPUFm7Fg4/3HexnH22X8JWRFKm8V6StJQuDD1jhg/tjh3hj3+Enj39wlnvv+/HkCvMRdKmFrokLeEVFLduhZde8v3j770He+wBv/yln83ZsWP2CxWpZxTokrQ6V1Bcs8Z3qTz0EKxaBV26+LXIL7oIdtstx9WK1B/qcpGkxVtB8bb9K+HnP/erHf7ud3DooTB5MixYAFddpTAXyTK10CVpsYtnfb52I2eumsO1/32DVrf/G5o0gYsv9rM5Dzoo5EpF6hcFuqRkQKfdGLDrB/DSA7BkCXToAHfd5WdztmgRdnki9ZICXZLz8cd+Nufo0bBxIxxzDAwfDv37+0lBIhKatP4Hmllz4HHgEMABP3fO/TsThUkecQ6mTvWjVSZPhkaN/BDEIUPgiCPCrk5EAuk2qUYAbzjnzjCzRkCTDNQk+WLTJhg71s/mnD8f9tkHbr0VLr/c3xaRvJJyoJvZ7sCxwEAA59xWYGtmypJQLVvmLyDx2GOwbp1vhT/9NPzsZ7DrrmFXJyJxpNNC7wSsAZ4ys8OAWcAQ59w3GalMcss5+Ne/fLfK+PH+/mmn+W6Vo4/WtTlFCkA6gd4AOAIY7Jx718xGANcDv4t9kpkNAgYBdOjQIY3dSTxpLZS1ZYu/uPKIETBrFjRvDr/+tR83vt9+2S1cRDIqnUBfAaxwzr0b3H8JH+g7cM6NAkYBlJSUuDT2JzVIeaGs1au/m825erUfM/7ww3DBBdC0aS5KF5EMSznQnXOfm9lyM+vqnFsInADMz1xpkkjLu7aFsmoM9NmzfWv8uef8Wit9+/r1VU48Ud0qIgUu3VEug4FnghEui4GL0y9JIPGWd7yFskrLytn/+tdo07yY607oTP9lM32Qv/22b4FfdhkMHgxdu2b/YEQkJ9IKdOfcHKAkQ7VIjERb3vEWygJotnkjJ08ZT8mwSbD+C7/C4d13wyWX+L5yEYkUTe3LU4kuUTu0T9cdWvIAnb9azsBZr3L6vDdpUrGFGe0P4cFTr+LO0TdBUVH1lxSRiFCg56k6l6gNVLXW7359AZ3n/IuLZ06k15JZbClqwMSDevFUST/m79MJA+5UmItEmgI9T9XU8i5uWMTQPtX6vDduZMC/X2HAmPth4UK+bLYn9xxzHs8e3pevmn7XrVL9jUBEokeBngOpjBOPXaK2xu9butRfNOLxx6GsDEpKYNw4ph94FI+/urDuNwIRiRwFepalc0HlAd3b7vgc5+Ctt/xolQkT/DDD00/3szl/+EMwoz/gGjZKfaKRiBQsBXqWJT1OvCZbtsDzz/sgnz0b9twTrrsOrrzSXx2omp3eCESkXtAl6LIs4Qsq1+Tzz+GWW/zFIwYOhC1bmH3TMI4fMpb93TEc/cwnTJhdmtmCRaRgKdCzLN7JyNpOUv79mdd5o/uJbG3bnu233cbnXQ+FqVOZMHYK59phLN7kcHzXfaNQFxFQoGddvAsq73SScts2ePFFvur+Pxx3/skcM386z3Tvy3GXPcpxPa9lwl4HMfyvH8ftvhERUR96ltU5WmXtWr/u+MiRsHw5m/dszf8dfxkvHtqbDbsGi2QFoZ1W942IRJ4CPQdqPEk5f76/EtCYMVBeDscdBw8+yLHvQOUuO08AqnozSGSykYjUTwr0LNpp/PmJBzLg87l+tMrUqf7qP+efD9dcA4ceCsC+86fFDe2EJxuJSL2kQM+S2PHnTbds4oQ3X+Xw4ZNgbSm0aQN33AGDBkHLljt8X22hXWf3jYjUawr0LBk+ZSF7rSll4KxXOWvuVHbfuonZrbvyp1OHMqdHb37dtxsDqoU51N3nrjHmIhKPAj3TnIN//pP/ffImei96j+1mTO56DE+V9GNOm6BrZENFrbNFFdoikgoFeqZs3gzPPuv7x+fOpUeT3Xn4yDMY2/1kVjfbuSWe9GxREZE6KNDTtXKlvy7no4/Cl1/C974Hjz/OWwf3ZOTkRTuNG9/hWzXcUEQySIGeqvfe863xF16Ayko49VR/bc5evcCMfsD2xsUMn7Iw7hWFNNxQRDJJgZ6MigoYPx7uuw9mzIBmzeDqq/1X5847Pb2qL7z6ioug4YYiknkK9ER89RWMGuVnc5aWwgEH+ElBAwf6UK+DhhuKSC4o0Gszb57vVhk3zp/07N0bHnkETj4ZdkluGRyNXBGRbFOgV7d9O7z2mg/yN9+Exo3hwgv9bM5u3cKuTkQkLgV6la+/hqeeggcegE8/hXbt4M474bLLYK+9wq5ORKROCvRPP/Uh/uSTsGEDHHUU/OEP8NOfQsOGYVcnIpKw+hnozsG0ab5bZdIkaNAAzjrLX5vzBz8IuzoRkZTUr0AvL4dnnvFBPm8etGoFN98MV1wBrVuHXZ2ISFrSDnQzKwJmAqXOuZ+kX1IWrFjhZ3OOGuWHIB52mO8vP/tsf9JTRCQCMtFCHwIsAHbPwGtl1owZfhLQyy/70Sv9+/tulWOPBbOwqxMRyai0rilqZu2AU4DHM1NOBmzd6hfJ6tEDfvhDeOMNP+Rw0SI/y7NnT4W5iERSui30+4DrgLjTJc1sEDAIoEOHDmnurhZr1nw3m3PVKujSBR58EC66CHbbLeWX3emqQ5rhKSJ5KuVAN7OfAF8452aZWa94z3POjQJGAZSUlLhU9xfX3OCSbs88A1u2wEknwRNPQJ8+Sc/mrK76GiylZeW1rmMuIhKmdBLvaKCfmX0GPA8cb2bjMlJVXSorYcIEf2Hlww6D55+Hiy/2F16eMgX69k07zMGvvVJ9+duqdcxFRPJNyi1059wNwA0AQQv9N8658zNUV83Wr/cTgB54AJYsgQ4d4K674NJLoUWLjO8u3nrlWsdcRPJRYYxD/+QTv7rh6NGwcSMcc4wP8gED/KSgLGnTvLjGtcy1jrmI5KP0+yUA59w/sjoG/dJL/QnP006DWbPg7bfhjDOyGuYAQ/t0pbhh0Q7btI65iOSrwmihP/SQXyBr331zulutYy4ihaQwAj3EZWu1jrmIFIqMdLmIiEj4FOgiIhGhQBcRiQgFuohIRCjQRUQiQoEuIhIRCnQRkYhQoIuIRIQCXUQkIhToIiIRoUAXEYkIBbqISEQo0EVEIkKBLiISEQp0EZGIKIz10PPEhNmlutiFiOQtBXqCJswu5YbxH1JeUQlAaVk5N4z/EEChLiJ5QV0uCRo+ZeG3YV6lvKKS4VMWhlSRiMiOFOgJWllWntR2EZFcU6AnqE3z4qS2i4jkmgI9QUP7dKW4YdEO24obFjG0T9eQKhIR2ZFOiiao6sSnRrmISL5SoMcRb4iiAlxE8lXKXS5m1t7M/m5mC8zsIzMbksnCwlQ1RLG0rBzHd0MUJ8wuDbs0EZG40ulD3wZc65w7CDgSuMrMDs5MWeHSEEURKUQpB7pzbpVz7v3g9gZgARCJ/ggNURSRQpSRUS5m1hHoDrxbw2ODzGymmc1cs2ZNJnaXdRqiKCKFKO1AN7PdgJeBXzrnvq7+uHNulHOuxDlX0qpVq3R3lxMaoigihSitUS5m1hAf5s8458ZnpqTwaYiiiBSilAPdzAx4AljgnPtT5krKDxqiKCKFJp0ul6OBC4DjzWxO8HVyhuoSEZEkpdw/RaNNAAAF9klEQVRCd869A1gGaxERkTRoLRcRkYhQoIuIRIQCXUQkIhToIiIRoUAXEYkIBbqISEQo0EVEIkKBLiISEQp0EZGIUKCLiESEAl1EJCIU6CIiEaFAFxGJCAW6iEhEKNBFRCJCgS4iEhEKdBGRiFCgi4hEhAJdRCQiFOgiIhGhQBcRiQgFuohIRCjQRUQiQoEuIhIRCnQRkYhQoIuIRERagW5mPzazhWa2yMyuz1RRIiKSvJQD3cyKgJFAX+Bg4BwzOzhThYmISHIapPG9/wMscs4tBjCz54H+wPxMFJZLE2aXMnzKQlaWldOmeTFD+3RlQPe2YZclIpKUdLpc2gLLY+6vCLbtwMwGmdlMM5u5Zs2aNHaXHRNml3LD+A8pLSvHAaVl5dww/kMmzC4NuzQRkaSkE+hWwza30wbnRjnnSpxzJa1atUpjd9kxfMpCyisqd9hWXlHJ8CkLQ6pIRCQ16QT6CqB9zP12wMr0ysm9lWXlSW0XEclX6QT6f4ADzWx/M2sEnA1MzExZudOmeXFS20VE8lXKge6c2wZcDUwBFgAvOOc+ylRhuTK0T1eKGxbtsK24YRFD+3QNqSIRkdSkM8oF59xkYHKGaglF1WgWjXIRkUKXVqBHxYDubRXgIlLwNPVfRCQiFOgiIhGhQBcRiQgFuohIRCjQRUQiwpzbabZ+9nZmtgZYmrMdJq8l8GXYRWSIjiU/ReVYonIcUBjHsp9zrs61U3Ia6PnOzGY650rCriMTdCz5KSrHEpXjgGgdi7pcREQiQoEuIhIRCvQdjQq7gAzSseSnqBxLVI4DInQs6kMXEYkItdBFRCJCgS4iEhH1MtDN7MdmttDMFpnZ9TU8vquZ/Tl4/F0z65j7KhOTwLH82szmm9lcM3vTzPYLo85E1HUsMc87w8ycmeXlULNEjsPMzgp+Lx+Z2bO5rjFRCfx9dTCzv5vZ7OBv7OQw6qyLmT1pZl+Y2bw4j5uZ3R8c51wzOyLXNWaEc65efQFFwKdAJ6AR8AFwcLXnXAk8Etw+G/hz2HWncSzHAU2C21cU8rEEz2sGvAXMAErCrjvF38mBwGygRXB/77DrTuNYRgFXBLcPBj4Lu+44x3IscAQwL87jJwOv46+VfCTwbtg1p/JVH1vo/wMscs4tds5tBZ4H+ld7Tn/g6eD2S8AJZlbTRbHDVuexOOf+7pzbFNydgb/2az5K5PcCcBtwF7A5l8UlIZHjuAwY6ZxbB+Cc+yLHNSYqkWNxwO7B7T3I0+sKO+feAtbW8pT+wBjnzQCam1nr3FSXOfUx0NsCy2Purwi21fgc5y+1tx7YKyfVJSeRY4l1Cb4Vko/qPBYz6w60d85NymVhSUrkd9IF6GJm081shpn9OGfVJSeRY7kVON/MVuCvXjY4N6VlXLL/l/JSfbxiUU0t7epjNxN5Tj5IuE4zOx8oAXpmtaLU1XosZrYLcC8wMFcFpSiR30kDfLdLL/wnprfN7BDnXFmWa0tWIsdyDjDaOXePmf0QGBscy/bsl5dRhfJ/vlb1sYW+Amgfc78dO39M/PY5ZtYA/1Gyto9rYUnkWDCz3sBNQD/n3JYc1Zasuo6lGXAI8A8z+wzfzzkxD0+MJvr39YpzrsI5twRYiA/4fJPIsVwCvADgnPs30Bi/2FWhSej/Ur6rj4H+H+BAM9vfzBrhT3pOrPacicBFwe0zgGkuOHOSZ+o8lqCb4lF8mOdrXy3UcSzOufXOuZbOuY7OuY748wH9nHMzwyk3rkT+vibgT1ZjZi3xXTCLc1plYhI5lmXACQBmdhA+0NfktMrMmAhcGIx2ORJY75xbFXZRSQv7rGwYX/gz2h/jz+DfFGz7P3xAgP+jfBFYBLwHdAq75jSO5W/AamBO8DUx7JpTPZZqz/0HeTjKJcHfiQF/AuYDHwJnh11zGsdyMDAdPwJmDnBS2DXHOY7ngFVABb41fglwOXB5zO9kZHCcH+br31ZdX5r6LyISEfWxy0VEJJIU6CIiEaFAFxGJCAW6iEhEKNBFRCJCgS4iEhEKdBGRiPj/DSDn6lW+q9IAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f458080c898>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_test = np.linspace(-0.1, 1.1, 500)[:, None]\n",
    "\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.metrics import mean_squared_error\n",
    "model = LinearRegression()\n",
    "model.fit(X, y)\n",
    "y_test = model.predict(X_test)\n",
    "\n",
    "plt.scatter(X.ravel(), y)\n",
    "plt.plot(X_test.ravel(), y_test,c='r')\n",
    "plt.title(\"mean squared error: {0:.3g}\".format(mean_squared_error(model.predict(X), y)));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have fit a straight line to the data, but clearly this model is not a good choice.  We say that this model is **biased**, or that it **under-fits** the data.\n",
    "\n",
    "Let's try to improve this by creating a more complicated model.  We can do this by adding degrees of freedom, and computing a polynomial regression over the inputs. Scikit-learn makes this easy with the ``PolynomialFeatures`` preprocessor, which can be pipelined with a linear regression.\n",
    "\n",
    "Let's make a convenience routine to do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.linear_model import LinearRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll use this to fit a quadratic curve to the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_poly = PolynomialFeatures(degree=2).fit_transform(X)\n",
    "X_test_poly = PolynomialFeatures(degree=2).fit_transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XeYFFXWx/HvAQYdEEEECYMCJhQT6BgQA2sC1wAqrogYENPqa1pkFzOiLqyoKOuuijlgQEUwoBjAFXENKCoCYgKBgRVUUNBRYbjvH7cGmqF7pns6VIff53n6menq6q5TE07fPvfWveacQ0REcl+dsAMQEZHUUEIXEckTSugiInlCCV1EJE8ooYuI5AkldBGRPKGELhIHM3Nmtn3YcYhURwldJE+Z2aFm9pmZ/WJmU8ysbTX77m9m75nZSjP7xMwOiHjsKDN7y8xWmNn/zOweM2uUmbOQRCihS8Exs7ohHrtePNsSfY0o+zQDxgFXA02B6cCTMfZtCjwHjACaADcBz5vZFsEujYEbgNbAzkCbYF/JMkroecrM5pvZoKC19bOZ3WdmLczspaAV9lrEPyxmtp+ZvR20wj42s24Rj/U3sznB8742s3MjHutmZovMbKCZLTWzJWbWv5q4zgheY6WZzTOzU4Ltdc3sZjP7Lnj8gqDMUS/ifA6LeJ0hZvZoxP2ngtbjj2b2ppntEvHYg2Z2p5lNNLOfgT+Y2SbB8RaY2bdmdpeZFUc8Z1BwLovN7MwaftaNg5/vEjMrM7MbKt80gvOdZmYjzewHYEiMbXXM7Coz+yb4OT5sZo2D12gX/CwGmNkCYHJ18QSOB2Y5555yzv0KDAH2MLOdouy7P/BtsG+Fc+5RYFnwGjjnHnPOveyc+8U5txy4B+gaRwySYUro+e0E4HBgR+AY4CXgCqAZ/nd/EYCZlQAv4lthTYHLgGfMrHnwOkuBo4HNgf7ASDPbM+I4LfGtuBJgAPCvyDeLSmbWEBgFHOmca4RPJB8FD58dHKMzUAr0TvBcXwJ2ALYCPgTGVHm8L3Aj0Ah4C/gH/ufSCdg+iP2aIM4ewc/g8OA1D6N6DwFrgtfpDBwBnBXx+L7A10FsN8bYdkZw+wOwLbAZcEeV4xyMbyF3D+L8xMz6xohpF+DjyjvOuZ+Br4LtVVlwq7pt1xivfRAwK8ZjEibnnG55eAPmA6dE3H8GuDPi/oXA+OD7vwGPVHn+JOD0GK89Hrg4+L4bUA7Ui3h8KbBflOc1BFbg32iKqzw2GTgv4v4RgKt83eB8Dot4fAjwaIz4mgTPbRzcfxB4OOJxA34GtovY1gWYF3x/PzA84rEdg9fbPsqxWgC/RZ4PcDIwJfj+DGBBledE2/Y6cH7E/Q7AaqAe0C44/rYJ/P7vizyHYNs04Iwo+24Z/F5OBoqA04G1wN1R9j0cWA7sGPbfuG4b39RCz2/fRnxfHuX+ZsH3bYETg3LLCjNbARwAtAIwsyPN7B0z+yF47I/4Vn6l751zayLu/xLx2us430o8CTgPWGJmL0aUAFoDCyN2/ybekwzKNcPN7Csz+wmf/KkSY+RrNwcaAB9EnO/LwfZEY2mLT4JLIl7rbnzLO9qxY21rXeU43+CTeYsaXieWVfhPVJE2B1ZW3dE59z3QE/gL/m+kB/AasChyPzPbD3gM6O2c+zyBWCRDEuqMkby1EN9CP7vqA2a2Cb51fxowwTm32szGs/FH9Lg45yYBk4J69Q34euyBwBJg64hdt6ny1J/xSbhSy4jv++IT0mH4ZN4Y34qMjDFyWtHv8G9ouzjnyqKEWVMskRbiW+jNqrypRYo2pWnVbYvxbw6Rx1yDT7BtqnmdWGbhW9rAunLXdsQolTjn/gPsHexbD1+euSXi+Z3xHadnOudeTyAOySC10AXgUeAYM+setHY3DTo72wD1gU3wnWRrzOxIfDkkYUGn7LFBcvkN34qsCB4eC1xkZm2C+vvgKk//COhjZkVmVrXG3ih4ve/xSf/v1cXhnFuLfyMZaWZbBbGVmFn3iFjOMLOOZtYAuLaa11oCvALcYmabB52b25nZwTX8OKp6HLjUzNqb2WbBOTxZzZtETZ4FdjWzE8xsU3z/wCfOuc+i7WxmnYOf7ebAzcCi4M0XM9sV/wnmQufc87WMRzJACV1wzi3Et3CvwCfuhcAgoI5zbiW+83QsvtXbF99Sq406wEB8a/QHfCff+cFj9+Dr9h/jOzXHVXnu1fgW5nLgOvxH/0oP40sUZcBs4J04Yvkb8CXwTlCmeQ1ft8Y59xJwG76u/yU1jyo5Df/GNzuI72mCclUC7gceAd4E5gG/4vs5YjKzWZWjhKpyzi3D91XcGMS0L9An4rl3mdldEU/5K/6Ty8Ig9uMiHhuIL0fdZ2argps6RbOQBR0dIlnFzNrhE1tREq1UkYKiFrqISJ5QQhcRyRMquYiI5Am10EVE8kRGx6E3a9bMtWvXLpOHFBHJeR988MF3zrnmNe2X0YTerl07pk+fnslDiojkPDOL68pplVxERPKEErqISJ5QQhcRyRNK6CIieUIJXUQkT2j6XJEcN35GGSMmzWXxinJaNylmUPcO9OpcEnZYEgIldJEcNn5GGZePm0n5aj8LcdmKci4fNxNASb0AqeQiksNGTJq7LplXKl9dwYhJc0OKSMKkhC6SwxavKE9ou+Q3JXSRHNa6SXFC2yW/KaGL5LBB3TtQXFR3g23FRXUZ1L1DSBFJmNQpKpLDKjs+NcpFQAldJOf16lyiBC6ASi4iInlDCV1EJE8ooYuI5AkldBGRPKFOURFJKc0tEx4ldBFJGc0tEy6VXEQkZTS3TLhqTOhmdr+ZLTWzTyO2NTWzV83si+DrFukNU0RygeaWCVc8LfQHgR5Vtg0GXnfO7QC8HtwXkQKnuWXCVWNCd869CfxQZXNP4KHg+4eAXimOS0RyUNhzy4yfUUbX4ZNpP/hFug6fzPgZZRk5braobadoC+fcEgDn3BIz2yqFMYlIjgpzbhl1yGZglIuZnQOcA7DNNtuk+3AiErKw5paprkO2UBJ6bUe5fGtmrQCCr0tj7eicG+2cK3XOlTZv3ryWhxMRqZ46ZGuf0J8DTg++Px2YkJpwRERqRx2y8Q1bfBz4L9DBzBaZ2QBgOHC4mX0BHB7cFxEJTdgdstmgxhq6c+7kGA8dmuJYRKTAJTNtgBb70KX/IpIlUjFKpdAX+9Cl/yKSFTRtQPLUQhfJEfk+i6FGqSRPCV2kBtmQSAvhopnWTYopi5K8C2mUSrJUchGpRmUiLVtRjmN9Is30JeWFUI7QKJXkKaGLVCNbEmnS5YiKClizxn9duxacS2F0qdGrcwnDjt+NkibFGFDSpJhhx++WN59AMkElF5FqZEtdt3WTYsqW/0LT8p9otfI7Wq78jlYrv6fd2p/hklfh++/97Ycf4Mcfobx8w9uaNdFfuF49aNgQGjTwXytvTZtC8+b+1qyZ/9qyJWyzjb9ttllKzitaOWva4ENS8tqFSAldpBoZr+uuXQsLFsDnn8Pcuf7r558zafZcipYsZpOK1Rs/591GsOWW/ta0KWy9tU/QxcXrb5tuCma+ZR55W70afv7Z3375xX9dtQoWLoQZM2DZMvj9942P2bQptG3rk/uOO8LOO8NOO/mvTZrEdaqF0C+QaUroItUY1L3DBkkHUljX/flnmDkTPvpo/e2TT3yLulKjRtChA5t13Y8v6jfhheX1mFuvMWtal9C757706LYb1K+ffCyxOOcT/LJlsGSJf7P55ht/W7AAvvgCXnppw6TfogXssgvsuae/7bUXbL891NmwwqvJtFJPCV2kGim7+tA5+PJLePvt9bdZs9bXsrfYAjp1gnPP9clwxx39rUUL37IGdgAujfLSaR2FY+bfVBo1gm23ha5dN95nzRqYPx/mzIHPPvNfZ86EUaPWJ/rNNoPOnaFLFzjgANh//6iffICY26Vm5jLYOVJaWuqmT5+eseNJfsmG4YNxc84nttdf97dp0+C77/xjjRv7xLbvvr4F26mTL5MEiTsRVcsW4D9BZEVn4urVMHu2L918+CG8/z588IHfDnyx5da836Yj7229K2+168R3Df1KlnXN+GrYH8OMPOuY2QfOudIa91NCl1yQ1Ymr0v/+B5Mm+QT+2mu+RAG+ZXvQQbD//v62884blR9qq+vwyVFbtCVNitPWuZjUG2t5OUyfDm+9xeR7x1FaNofNf/sZgNlbtefNdp2Z2n5Pxjw40Nf9BVBClzwTRuKqkXPw6afw3HP+9t57fnuzZnDooXDYYf5r+/ZpC6H94BeJ9h9swLzhR6X8eKl8Y+06fDJLflhFx6XzOGjehxw4fwZ7LZpD/bVrfEfu4YdDz55wzDF+lE0Bizehq4YuOSFbhg+ydi288w6MHQsTJvjaMcDee8P118NRR8Eee6SsBV6TTI/CSWVHZmWH86ctt+fTltvz7y5/Ykv3O3e0WUmXz99f/0ZZp47/ZNOrFxx3nP/EI1HpwiLJCaEuXuCcrwH/9a++td21K9x1F+y6K4weDYsX+9b5VVf5jr8MJXPI/NWVqXxjjXYh0dV99qHLhafBP//p3yxnzICrr/YjbS67DLbbzvc//POf8O23yZ1MHlLJRXJCKDX0r7+Ghx6Cxx/3w/Pq1YMjjoA+fXwpYPPN03PcOFXWsstWlFPXjArnKElzZ3Gopa/58+Gpp+Cxx/wQzzp1fEmrb1/fcm/cOL3HD1G8JRe10CUnVLbmmhQXrdu2aVEa/nzLy2HMGDjkEN8avP56PwJl9Gjf6fnii3DqqVmRzCvnmAGocG5dyzydncSp/kQwfkYZXYdPpv3gF+k6fHL1c+S0aweDBvlW+6xZcPnlfiho//7QqhWccQa89VZWTmuQKUroklN+W7N23ffLf1mduomyZsyACy6A1q2hXz/fGrz+en/xzOuvw9ln+ysxs0RYc8ykcr6VpCY+69gRbrgBvvrKj+k/9VR45hk48EA/jv/WW9cPEy0gKrlIzkj5x/3ff/dJYNQo39G5ySZwwgkwYAB065bRWniiUjW6Jcyx/Sn/fa5a5Tur77nH/z6LiuD44+Hii2G//Wo1zj9bqOQiGZPQx+YkpKxD7ttvYehQPxdJ375+Uqvbb/fjxivLLVmczCE1ncRhTw2c8pFLm20GZ54J//2vn0Lhz3+Gl1/2I2T228/3hayOMhdOHsnuv1rJeplMCkknsY8/htNO8xNKXXutv0Jz4kR/ufpFF/nL73NEKmrZYU8NnNaRS7vtxvgz/sphl47hmsPPY8FXZf7Nu317GDbMv4nnISV0SUomk0Ktk9jUqX58eKdO8Oyzfr6Uzz7zk0odeWTWt8ajSUUtO+yx/ekcclnZ0PiyHB7e82gOHnAn5/a5jqVbbwdXXOE7ui++GBYtSvpY2UQXFuWZTNdEM5kUEpooyznf+h42zM+j0qyZ70Q7//ycaolXJ9kV7sNe8i1lE59FUbWh4awOk9ruxad7HMC0e7aCW26Bf/8b7rwTTj8d/vY3PyNkjlNCzyNhzC+d6aRQYxJbuxbGjfMjVD75xJdXRo3yHZ0NGqQlplyV1qmB45Tsm1Is1TY0dt0VHngAhgyBESPg3nvh/vvhpJP8UMjddkt5PJmSe581JaYwaqJZsw6kc/D8837u7RNPhN9+gwcf9OOUL7xQyTyKfF7yLa76fNu2cMcdfojqZZf5v5/dd/eJfW5urtWqhJ5HwqiJhp4UnINXXvGjGI49FlauhIcf9heenH66H7omMfXqXMK0wYcwb/hRTBt8SF4kc0iwodGyJfzjH37Rjquu8hePdezoR8x8802GIk4NjUPPI1k5I2E6vf02DB7sOz233hquuUZJvIqcmkM+xWp97kuXwvDhvsa+di2cd57vSG3ZMv1Bx5CR6XPN7FLgLMABM4H+zrlfY+2vhJ5eOTFneCp89ZXvxHrmGf9PdtVVcNZZ/sIgWadg/h7SZdEi3xdz333+b+uyy/wEbQ0bZjyUtF9YZGYlwEVAqXNuV6Au0Ke2ryfJC738kW4//ACXXuoXiHj5ZbjuOl8jv+CCgk3m1V3UFfY485zXpg3cfbcf4nrMMf5itB128H0za9fW+PQwJDvKpR5QbGargQbA4uRDkmSka9RAqH77Df71L99a+uknX9scOtRPyFTAahrVFPY487yx/fbwxBN+3Pqll/rJwEaNgpEj4eCDw45uA7VuoTvnyoCbgQXAEuBH59wrqQpMBPBLuu22GwwcyLc778GpF95N+y170fWhORm7RD1b1dQCD3UO+XzUpYufVuCxx/yVpt26+blivv467MjWSabksgXQE2gPtAYamlm/KPudY2bTzWz6smXLah+pFJZvvvH/LD16APD2Px+m26GDmbppq1DmHclGNbXAs2ZIaT4xg5NP9mWYG2+EV1/1sztefz38GrP7MGOSGbZ4GDDPObfMObcaGAfsX3Un59xo51ypc660eYGvC5ivUjo512+/+X+Uyjr53/8OM2cyaFWJ6sFV1NQCz/s+lTAVF/uRL5995hc7ueYa/0ly0qRQw0qmhr4A2M/MGgDlwKGAhrAUmJRenfrKK76D88sv/TS2t97qr/Qk/HlHslE8V3rmZZ9KNikp8fX1s87yf7s9ekDv3r6+3qZNxsNJpob+LvA08CF+yGIdYHSK4pIckZKRFN9952dB7N7df6R9+WV4+ul1yRxUD45GLfAscthhfqqJG26AF16AnXbySb2ioubnppAuLJKkJLXQgnPrRw8sX+4vErrySth00412DXtMdbSLVCA9E0tJjps3z0838eKLsPfefq6Y3XdP6iXjHYeuybkkKbWenGvBAr8AwcSJsM8+fpm3aiZFSufMfDWJVlYa9NTHYLC6wq3blu6J0CRHtG/v54V58kk/z/5ee/kLkq6+OmpjJZXUQpekJNxyXrsW7rrLX+m5dq3vAL3wQqhbd+N9s0SsKRWiydtpFqR2vv8eBg6ERx+F6dP9nPy1oBa6ZERCLeeFC/1FQa+9Bkcc4a/Ca9cuoeOFMTdJIh2vhdxJK1FsuaW/svTqq2G77dJ+OCV0SVpNIynGf7iImcPv4OLn7qCeq2DulcPpfP1fE160N4z53iF2WSnWviIbyUAyB02fK2k28fWP2azvn7j6qX8wp3k7uve/g762B+M/qn6WiGhj28OamyTaBTpFdYyiuhu+IemiHQmbWuiSPhMm0OWUM2jw6ypu7HYm9+3dk7V16kKQhGO1qmO1xKsm80rpLnPEKitF26YOUQmTErqkXnk5/OUvcNddlLXYjr+cdAOfN2+3wS7VJeFYLfG6ZlRE6cTPRJkjVllJCVyyiRK6pNann0KfPn7FoEGDuGDzQ/lm1ZqNdqsuCcdK9hXOUVxUN9Q1MEWymWrokhrO+RXU997bX/k5aRLcdBOXHrVrwhNExUr2lVdC6spIkejUQpfk/fCDn8vi2Wf95fsPPQQtWgC1uyCoujlKNDeJSGxK6JKcd9+FE0+E//0Pbr7ZLwBQZ8MPfokm4TCvChXJZUroUjvOwR13+Kvg2rTxCzaX1nghW9zUEhdJnGrokrhVq6BvXz9PRY8e8MEHKU3mIlI7SuiSmNmzfcfn2LF+8Ynx42GLLcKOSkRQyUUSUTmRf8OGfj6WP/wh7IhEJIJa6FKzigoYNMivpdipE3z4oZK5SBZSC12qt3y5v1Cocnm4kSOhqCjsqEQkCiV0iW3OHL8A7vz5MHo0nH32uofCmMZWRKqnhC7RvfCCH8lSXAyTJ8MBB6x7KKxpbEWkeqqhy4acg2HD4NhjYYcd/CorEckcUrQwtIiknFrost6vv8KAAfDYY74D9N57oUGDjXaLNXmWVusRCZda6OJ99x0cdphP5sOGwZgxUZM5xJ48S6v1iIRLCV3g889hv/18eWXsWBg8uNrl4aKt4KNpbEXCp5JLoZs6FXr1grp1YcoU6NKlxqdo8iyR7KSEXsjGjIEzz4T27WHiRNh227ifqsmzRLKPSi6FyDkYOhT69fMt8rffTiiZi0h2Ugu90KxZA+eeC/ffD6ee6key1K8fdlQikgJJtdDNrImZPW1mn5nZHDOruQAr4fnlFzj+eJ/Mr7rKryykZC6SN5Jtod8OvOyc621m9YHo49wkfMuXwzHH+PLKHXf4eVlEJK/UOqGb2ebAQcAZAM6534HfUxOWpFRZmV/r84sv4Mkn/ZJxIpJ3kim5bAssAx4wsxlmdq+ZNUxRXJIqn30G++8PCxbASy8pmYvksWQSej1gT+BO51xn4GdgcNWdzOwcM5tuZtOXLVuWxOEklvEzyug6fDLtB79I1+GTGT+jzD/w3nt+HpZff4U33oBDDgk1ThFJr2QS+iJgkXPu3eD+0/gEvwHn3GjnXKlzrrR58+ZJHE6iqZz5sGxFOY71Mx9Ou+sJn8AbN4Zp02DPjX41IpJnal1Dd879z8wWmlkH59xc4FBgdupCk3jmHI828+H+c96mdPhw2HknvzBFy5aZDFtEQpLsKJcLgTHBCJevgf7JhyQQ/5zjVWc4PGrOVG574WZmt9iWPd54A5o2zVjMIhKupMahO+c+CsopuzvnejnnlqcqsEIX75zjkTMc9p75GqOeH8GM1h04rc/faX/TfzesqYtIXtOl/1kq3jnHK2c+7Pfhi9w88Tamtd2D008cyo/1izeoqSupi+Q/JfQsFe+c4706l/DUyre44dU7eXX7fTm39zWU1990g320mpBIYVBCz1JxzTnuHAwZwq633wh9+nD47Kn8Wrco6utpNSGR/KeEnqV6dS5h2PG7UdKkGANKmhQz7Pjd1neIOgfXXgvXXeenwH30USgq0mpCIgVMsy1mQDzDD6OJOed4ZTK//no46yy4+26o49+bB3XvsMHoGNBqQiKFQgk9zeIdfhi3oMwSLZlHvqZWExIpPEroaVbd8MOEk2xlMh86FAYM2CiZV9JqQiKFSTX0NIt3+GFcgmT+fOmRbLtlT7re9IaGI4rIOmqhp1nrJsWURUne1XVSRq25T7gHhg7lmT2O4LJD/oyzOsmXb0Qkr6iFnmZxDT+MEG2yrYWXDIbrruOFvXpwWff/w9n6X5vGmItIJbXQ0yzRTsqqNfez3x3HhW8+ysQ9j+CiQ8/fIJlX0hhzEQEl9LSqWjoZeVKnGksjkcm534yJXPnG/Ty/04FccugFtNqiYcLlGxEpHCq5pEmsecpr6sSsTM7Hf/o6N7zyb17dfh8uPXogLZtulnD5RkQKixJ6msQ7W2JVg7p34Ngv/8uIibfzVts9+L+egynadJN1ZZpqrx4VkYJmzrmMHay0tNRNnz49Y8cLU/vBLxLrJ1tSXR194kTW9uzFzJIOnHz8ELbYagtdGCRS4MzsA+dcaU37qYaeJrGGK0I1V4tOmQInnECd3Xdjj8mTmd24cSZCFZE8oZJLmkSrd0faqPzy7rtwzDGw3XYwaZJfC1REJAFqoadJ5HDFWC31dSNa5syBP/4RWrSAV1+FZs0yFaaI5BG10NOoV+cSpg0+hJLqprRduBCOOAKKinwyb9Uqw1GKSL5QQs+AWMMNr9h3K+jeHX76CV5+GbbdNqQIRSQfqOSSAdGuFh180NYc9ZdT4euvfc28U6eQoxSRXKeEniEbTGm7ejUceyy8/z48/TQcfHC4wYlIXlBCz7S1a6F/f19iueceOO64sCMSkTyhGnomOQeXXQZjxsCNN/oVh0REUkQJPZNuuw1GjoSLLoLLLw87GhHJM0romfL00zBwIJxwgk/qZmFHJCJ5Rgk9E95+G/r1gy5d4JFHoq4DKiKSrKQzi5nVNbMZZvZCKgLKO59/7ke0bLMNTJgAxZq7XETSIxVNxYuBOSl4nfyzdCkceaRvkb/0ki7pF5G0Siqhm1kb4Cjg3tSEk0d++cW3zJcsgeef95NuiYikUbLj0G8D/go0SkEs+aOiAk45Bd57D8aNg333DTsiESkAtU7oZnY0sNQ594GZdatmv3OAcwC22Wab2h4uNFXXBY1rsYmBA2H8eBg1Cnr1ykygIlLwkim5dAWONbP5wBPAIWb2aNWdnHOjnXOlzrnS5s2bJ3G4zKvVuqB33gm33w6XXAIXXpixWEVEap3QnXOXO+faOOfaAX2Ayc65fimLLAskvC7oq6/6JH700XDzzRmIUERkPQ2IrsbimhamiPTZZ3DiidCxIzz2GNSNvVqRiEg6pCShO+fecM4dnYrXyiatq1uYItL33/tW+Sab+BEtjdRHLCKZpxZ6NWItTDGoe4f1G37/3V/Ov3AhPPsstG2b4ShFRDxNn1uNaAtTbDDKxTk4/3z4z3/g0Udh//1DjFZECp0Seg02WJiiqpEj4b774Mor/bhzEZEQqeRSWy+84Oc2790bhg4NOxoRESX0Wpk1C04+GfbcEx56SLMnikhWUCZK1PLl0LMnNGzoZ09s0CDsiEREANXQE1NR4VvmCxbAlClQUsMUACIiGaSEnoirroJJk+Duu6Fr17CjERHZgEou8Ro7FoYPh3PPhXPOCTsaEZGNKKHH4+OPoX9/3yofNSrsaEREolJCr8n33/spcJs08Qs9168fdkQiIlGphl6dNWvgpJNg8WKYOhVatgw7IhGRmJTQq/O3v8Hrr8P998M++4QdjYhItVRyieXxx+HWW/385v37hx2NiEiNlNCjmTULzjoLDjwQbrkl7GhEROKihF7VTz/B8cf7Oc2ffBKKisKOSEQkLqqhR3IOBgyAr76CyZOhVauwIxIRiZsSeqTbbvNDE0eMgIMOCjsaEZGEqORS6a23YNAgOO44GDgw7GhERBKmFjrAt9/Cn/4E7dvDAw+AWdTdxs8oi716kYhIyJTQ16yBPn1gxQp4+WVo3DjqbuNnlHH5uJmUr64AoGxFOZePmwmgpC4iWUEll6uugjfe8DMo7r57zN1GTJq7LplXKl9dwYhJc9McoIhIfAo7oU+YAP/4h59B8dRTq9118YryhLaLiGRa4Sb0efPg9NOhtNSPbqlB6ybFCW0XEcm0wkzov//u6+bg5znfdNManzKoeweKi+pusK24qC6DundIR4QiIgkrzE7RK66A996Dp57yI1viUNnxqVEuIpKtCi+hv/CCn5/l/POhd++lGAc1AAAIiElEQVSEntqrc4kSuIhkrVondDPbGngYaAmsBUY7525PVWBpsXChr5t36lTjpFsacy4iuSaZFvoaYKBz7kMzawR8YGavOudmpyi21FqzBk4+2dfPn3yy2rq5xpyLSC6qdaeoc26Jc+7D4PuVwBwge7PdtdfCtGl+vPmOO1a7q8aci0guSskoFzNrB3QG3k3F66XcK6/AsGF+JsW+fWvcXWPORSQXJZ3QzWwz4BngEufcT1EeP8fMppvZ9GXLliV7uMQtWQL9+sHOO8OoUXE9RWPORSQXJZXQzawIn8zHOOfGRdvHOTfaOVfqnCtt3rx5ModLXEUFnHIKrFrlx5s3aBDX0zTmXERyUTKjXAy4D5jjnLs1dSGl0E03wZQpcN99sMsucT9NY85FJBeZc652TzQ7AJgKzMQPWwS4wjk3MdZzSktL3fTp02t1vIS99x507eqXk3viiZhT4oqIZDsz+8A5V1rTfrVuoTvn3gKyM0uuXOk7P1u1grvuUjIXkYKQn1eKXnSRn3xryhTYYouwoxERyYj8m5xr7Fh48EE/X4vWBRWRApJfCf2bb+Ccc2DffeGaa8KORkQko/InoVdU+EUqKipgzBgoKgo7IhGRjMqfGvqwYTB1Kjz8MGy3XdjRiIhkXH600N95B4YM8ZNv9esXdjQiIqHI/YT+00/+atA2beDOOzVEUUQKVu6XXC65BObPhzffhMaNw45GRCQ0ud1CnzABHngALr/cXxUqIlLAcjehL10KZ58NnTtriKKICLma0J3zyfynn+CRR6B+/bAjEhEJXW7W0B94AJ57zq8LmsAsiiIi+Sz3Wujz5sHFF0O3br5DVEREgFxL6BUVcMYZUKeOn6+lTm6FLyKSTrlVchk50g9PfPBBaNs27GhERLJK7jRxZ86EK6+EXr3gtNPCjkZEJOvkRkL/7Tc/8VaTJjB6tK4GFRGJIjdKLkOGwMcf+5EtmV5oWkQkR+RGC93Mz3N+zDFhRyIikrVyo4X+97/7i4lERCSm3Gihg+rmIiI1yJ2ELiIi1VJCFxHJE0roIiJ5QgldRCRPKKGLiOQJJXQRkTyhhC4ikieSSuhm1sPM5prZl2Y2OFVBiYhI4mqd0M2sLvAv4EigI3CymXVMVWAiIpKYZC793wf40jn3NYCZPQH0BGanIrBMGj+jjBGT5rJ4RTmtmxQzqHsHenUuCTssEZGEJFNyKQEWRtxfFGzbgJmdY2bTzWz6smXLkjhceoyfUcbl42ZStqIcB5StKOfycTMZP6Ms7NBERBKSTEKPNrnKRjNoOedGO+dKnXOlzbNw6tsRk+ZSvrpig23lqysYMWluSBGJiNROMgl9EbB1xP02wOLkwsm8xSvKE9ouIpKtkkno7wM7mFl7M6sP9AGeS01YmdO6SXFC20VEslWtE7pzbg3wf8AkYA4w1jk3K1WBZcqg7h0oLqq7wbbioroM6t4hpIhERGonqQUunHMTgYkpiiUUlaNZNMpFRHJdbqxYlGa9OpcogYtIztOl/yIieUIJXUQkTyihi4jkCSV0EZE8oYQuIpInzLmNrtZP38HMlgHfZOyAiWsGfBd2ECmic8lO+XIu+XIekBvn0tY5V+PcKRlN6NnOzKY750rDjiMVdC7ZKV/OJV/OA/LrXFRyERHJE0roIiJ5Qgl9Q6PDDiCFdC7ZKV/OJV/OA/LoXFRDFxHJE2qhi4jkCSV0EZE8UZAJ3cx6mNlcM/vSzAZHeXwTM3syePxdM2uX+SjjE8e5/MXMZpvZJ2b2upm1DSPOeNR0LhH79TYzZ2ZZOdQsnvMwsz8Fv5dZZvZYpmOMVxx/X9uY2RQzmxH8jf0xjDhrYmb3m9lSM/s0xuNmZqOC8/zEzPbMdIwp4ZwrqBtQF/gK2BaoD3wMdKyyz/nAXcH3fYAnw447iXP5A9Ag+P7PuXwuwX6NgDeBd4DSsOOu5e9kB2AGsEVwf6uw407iXEYDfw6+7wjMDzvuGOdyELAn8GmMx/8IvIRfK3k/4N2wY67NrRBb6PsAXzrnvnbO/Q48AfSssk9P4KHg+6eBQ80s2qLYYavxXJxzU5xzvwR338Gv/ZqN4vm9AFwP3AT8msngEhDPeZwN/Ms5txzAObc0wzHGK55zccDmwfeNydJ1hZ1zbwI/VLNLT+Bh570DNDGzVpmJLnUKMaGXAAsj7i8KtkXdx/ml9n4EtsxIdImJ51wiDcC3QrJRjediZp2BrZ1zL2QysATF8zvZEdjRzKaZ2Ttm1iNj0SUmnnMZAvQzs0X41csuzExoKZfo/1JWKsQVi6K1tKuO3Yxnn2wQd5xm1g8oBQ5Oa0S1V+25mFkdYCRwRqYCqqV4fif18GWXbvhPTFPNbFfn3Io0x5aoeM7lZOBB59wtZtYFeCQ4l7XpDy+lcuV/vlqF2EJfBGwdcb8NG39MXLePmdXDf5Ss7uNaWOI5F8zsMOBK4Fjn3G8Zii1RNZ1LI2BX4A0zm4+vcz6XhR2j8f59TXDOrXbOzQPm4hN8tonnXAYAYwGcc/8FNsVPdpVr4vpfynaFmNDfB3Yws/ZmVh/f6flclX2eA04Pvu8NTHZBz0mWqfFcgjLF3fhknq21WqjhXJxzPzrnmjnn2jnn2uH7A451zk0PJ9yY4vn7Go/vrMbMmuFLMF9nNMr4xHMuC4BDAcxsZ3xCX5bRKFPjOeC0YLTLfsCPzrklYQeVsLB7ZcO44Xu0P8f34F8ZbBuKTxDg/yifAr4E3gO2DTvmJM7lNeBb4KPg9lzYMdf2XKrs+wZZOMolzt+JAbcCs4GZQJ+wY07iXDoC0/AjYD4Cjgg75hjn8TiwBFiNb40PAM4Dzov4nfwrOM+Z2fq3VdNNl/6LiOSJQiy5iIjkJSV0EZE8oYQuIpInlNBFRPKEErqISJ5QQhcRyRNK6CIieeL/AVGIYjNsBySHAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f45807ae278>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = LinearRegression()\n",
    "model.fit(X_poly, y)\n",
    "y_test = model.predict(X_test_poly)\n",
    "\n",
    "plt.scatter(X.ravel(), y)\n",
    "plt.plot(X_test.ravel(), y_test,c='r')\n",
    "plt.title(\"mean squared error: {0:.3g}\".format(mean_squared_error(model.predict(X_poly), y)));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This reduces the mean squared error, and makes a much better fit.  What happens if we use an even higher-degree polynomial?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEICAYAAABLdt/UAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJztnXecVNX5/9/PFnpZpMrSkSJgQVbFEkVFwApq7C2CEmOMyS8JCpbEfI2RSGKLFaLB2BUJCiigCBZAFAWWjiKCLEgRls6yy57fH2dmuTs7szvlTr3P+/Wa18y999xzz5nyuc885znPEWMMiqIoSuaQlewGKIqiKO6iwq4oipJhqLAriqJkGCrsiqIoGYYKu6IoSoahwq4oipJhqLArSjWIiBGRo5LdDkWJBBV2RckwROQcEVkpIvtEZJaItK+m7CwR2Soiu0RksYgMdhy7W0T2OB77RaRcRJo5yvQXka9FZK+I/CAiV8S7f0rNiE5QUryCiGQbYw5FeI4Buhhjvo3x2jnGmLKa9kVaR5AyzYA1wM3AZOAB4GfGmL4hyh8LLDfGlInIycCHQFdjzKYgZe8HzjDGnO3b7gHMBm4EPgAaA3nGmDXh9kmJD2qxZxgi8r2IjBCRQp8V9byItBSR90Vkt4h8KCJNHOX7ishcESn2WWz9HMduEpEVvvO+E5FfOo71E5ENIvIHEdkiIptE5KZq2vULXx27RWStiFzr258tIv8QkW2+47/2uT9yHP3p76jnfhF52bH9loj8KCI7ReQTEenpODZeRJ4RkfdEZC9wlojU9l1vvYhsFpFnRaSu45wRvr5sFJGhNbzXjX3v7yYRKRKRv4pItqO/c0TkURHZDtwfYl+WiNwrIut87+N/RaSxr44OvvdimIisBz6qrj0+LgWWGWPeMsYcAO4HjhOR7sEKG2MKHTcLA+QCbYP0VYDrgRcdu+8FnjPGvG+MKTPG/KSinhqosGcmlwHnAl2Bi4D3gbuBZtjP/A4AEckHpgJ/BY4A/gi8LSLNffVsAS4EGgE3AY+KyAmO67TCWmn5wDDgKedNw4+I1AeeAM4zxjQETgUW+Q7f4rtGb6AA+HmEfX0f6AK0AL4GXgk4fg3wINAQ+Az4O/Z9OR44ytf2P/naOcj3Hpzrq7M/1fMiUOarpzcwAGsp+zkZ+M7XtgdD7PuF73EW0AloADwZcJ0zgaOBgb52ForINSHa1BNY7N8wxuzFWvA9Q5RHRKaIyAFgPtYCXxCk2M+AlsDbjn19fecv8d3cXhaRI0JdR0kgxhh9ZNAD+B641rH9NvCMY/s3wCTf67uAlwLOnw7cGKLuScBvfa/7AfuBHMfxLUDfIOfVB4qxN5y6Acc+Am51bA/AWo45jv70dxy/H3g5RPvyfOc29m2PB/7rOC7AXqCzY98pwFrf6xeA0Y5jXX31HRXkWi2BEmd/gKuBWb7XvwDWB5wTbN9M4DbHdjegFMgBOviu3ymCz/95Zx98++YAv6jhvFzgPOD/VVPv+IB9B32fT1fsDelt4JVk/wb0YdRiz1A2O17vD7LdwPe6PXC5zw1TLCLFwOnAkQAicp6IfC4i233Hzsda/X5+MpV9vvscdVdgrNV4JXArsElEpjpcA62BHxzF14XbSZ8bZ7SIrBGRXViRIaCNzrqbA/WArxz9nebbH2lb2mPFcJOjruewlniwa4fa1zrgOuuwot6yhnpCsQf7D8tJI2B3dScZY0qNMe8DA0XkYucxn6vqciq7YcB+l/5jjFltjNkD/A37HVGSjAq7t/kBa7HnOR71jTGjRaQ21gL7B9DSGJMHvIe1eiPGGDPdGHMu9qaxEhjnO7SJyj7ddgGn7sWKsZ9WjtfXAIOxLpPGWAuXgDY6owO2YcWop6O/jY0x/ptRTW1x8gPWYm/mqKuRMcbp8ggWmRC4byP2JuG8ZhmVb8aRRDgsA47zb/jcYJ19+8Mhx1feyaXAdqybxklhhG1TEoQKu7d5GbhIRAb6rN86vkHRNkAtoDawFSgTkfOwbpKI8Q3eXuwTmRKsVemPTnkTuENE2vj88yMDTl8EXCUiuSIS6INv6KvvJ6z4/626dhhjyrE3lEdFpIWvbfkiMtDRll+ISA8RqQf8uZq6NgEzgH+KSCPfIGhnETmzhrcjkNeA/yciHUWkga8Pb5gIomUC+B/QS0QuE5E62PGDQmPMysCCItLd96+sru/9vQ44A/g4oOiNWJdWoIj/B7hJRDr53q+7gClRtltxERV2D2OM+QFr8d6NFfAfgBFAljFmN3aQ9U1gB9Y6fjfKS2UBf8Bap9uxg4G3+Y6Nw/r1F2MHPycGnHsf1oLcAfwFeNVx7L9Y10URsBz4PIy23AV8C3zuc998iPVr43NFPIb1+39LzVEoN2BvgMt97ZuAz40VAS8ALwGfAGuBA9hxkJCIyDLxRRUFYozZih3LeNDXppOBqxznPisiz/o3sWMWW7Cf/2+BK40xXzvK5wNnY9/rwGu94Ns/H/s5lOAbmFeSi8axKymFiHTAClxuDFarongatdgVRVEyjLCFXURe8E2gWBrk2B99EymaBTtXURRFSRyRWOzjgUGBO0WkLXZCx3qX2qR4GGPM98YYUTeMokRP2MJujPkEO/AVyKPAnWjYk6IoSkqQE8vJvokMRcaYxTaVRLVlhwPDAerXr9+ne/egqSuSy+LFkJcH7UMmw1MUJZGUldnfZdu20KJFzeWd7NgB330HPXvCpk2wZw8cc0x82pkgvvrqq23GmOY1lYsoKsYXsTDFGNPLF7c6CxhgjNkpIt8DBcaYbTXVU1BQYBYsCJaOIsm0bg0XXADjxtVcVlGU+LNpk/1dPvss/PKXNZd38uabcOWVsGwZjB4Nn31mhT6NEZGvjDEFNZWLJSqmM9ARWOwT9TbA1yLSqtqzUhkR0PBPRUkdSkvtc04MzgUP/qajfreMMUtw5MWIxGJPWbKyPPklUJSUpcw3hp6bG/m5NbiHM5lIwh1fA+YB3cTm4R4Wv2YlCREoL092KxRF8aMWe1SE/W4ZY66u4XiHmFuTbNQVoyiphVrsUaEzT52osCtKauGGxQ6e+12rsDtRH7uipBZ+i90NV4yHLHgVdifqY1eU1EJdMVGhwu5EXTGKklro4GlUqLA7UWFXlNRCLfaoUGF3oj52RUkt3Bo89Rgq7E7Ux64oqUUsFrsfDxprKuxO1BWjKKlFLBa70xXjsd+1CrsTdcUoSmqh4Y5RocLuRF0xipJa6OBpVKiwO1FXjKKkFhruGBUq7E5U2BUltVCLPSpU2J2oj11RUgsNd4wKFXYn6mNXlNRCwx2jQoXdibpiFCW10HDHqIhkoY0XRGSLiCx17BsjIitFpFBE/iciefFpZoJQYVeU1ELDHaMiEot9PDAoYN8HQC9jzLHAamCUS+1KDllZ6opRlFTCb7Hr4GlEhC3sxphPgO0B+2YYY3y3VD7HLmidvujgqaKkFiUl9rl27eS2I81w08c+FHjfxfoSj1rsipJaHDwI2dn2ES0eNNZcEXYRuQcoA16ppsxwEVkgIgu2bt3qxmXdR4VdUVKLkhKoVSu6c9UVEz0iciNwIXCtMaFvjcaYscaYAmNMQfPmzWO9bHxQYVeU+FNaCi+/DF9+WXPZgwdjd8N40GKPKepfRAYBdwFnGmP2udOkJKLCrijx5ze/geees4L99dfQo0fosm5Z7B4T90jCHV8D5gHdRGSDiAwDngQaAh+IyCIReTZO7UwMKuyZz7598PjjMGwYPP00HDiQ7BZ5i2+/taJ+2WV2++mnqy9fUuLewKmHXDNhW+zGmKuD7H7exbYkn6ysw3GzSuaxYweceSYsWQLNm8MLL8Dzz8OMGdC0abJb5w1efNH+zh5/3IYwvvEGPPGE3ReMgwejt9j9eMxaB515Whm12DOXsjK44gpYuRLeew+2bIF33oFly+Dyy/VzTxQTJ0K/fpCfD+eeC9u2wapVocvHYrF7yEIPRIXdiQp75vLss/Dhh/DMM3DeeXbfxRfDk0/CrFn2uBJfvv8eli+HCy+026efbp8/+yz0OWqxR4UKuxMV9sxk5074y1+spTh0aOVjw4ZZy3HkSNi+PejpikvMnm2fBwywz126QF6eHUANhVrsUaHC7kSzO2YmTzxh//L/4x9Vf+wi8MgjsHs3PPZYctrnFebNs0J+9NF2WwR69YKlS0Ofo+GOUaHC7kRTCmQepaXW/TJoEPTpE7xMr142SuPxx6G4OLHt8xLz5sHJJ1ceKO3Z045zhPrdabhjVKiwO1FXTOYxcSJs2gS33159uVGjYNcu+O9/E9Mur7Frl7XMTzml8v6ePW200o8/Bj/PDYvdj4dcMyrsTlTYM4+xY6FTp8MDpqHo0wdOPNHGWKepdTdpYRGnjf6IjiOnctroj5i0sCjZTTrMF1/Y9zVQ2Lt0sc9r1gQ/LxaL3U+afp6xoMLuRIU9s9i40Ua8XH996DhpJ7feaqM2qovSSFEmLSxi1MQlFBXvxwBFxfsZNXFJ6oj7vHnWYj755Mr7O3e2z9UJuw6eRowKuxMV9szijTestXZ1sLl1QbjySqhf3+YxSTPGTF/F/tJDlfbtLz3EmOnVxIgnknnzbOqAxo0r72/f3grwd98FP0/DHaNChd2JCntm8dpr1sXSrVt45evXh8GDYcKEwws8pAkbi/dHtD+hlJdbYQ90w4AV7bZt1WJ3GRV2JyrsmUNRkc0eePnlkZ131VU2nv3DD+PTrjjROq9uRPsTyqpVNtoomLADdOxoJy8Fww2LHTxntauwO1FhzxymTLHPF10U2XkDBthY69dfd79NcWTEwG7Uza28GEXd3GxGDAzz30o8mTfPPp96avDjbdrYG3Ew3EgCpmueehwV9sxh8mQbDeOfDBMutWvDpZfC//6XVpkfh/TO56FLjyE/ry4C5OfV5aFLj2FI7/xkN80Ke5Mm0LVr8OP5+XagO9hvLxaL3UNCHkhM+dgzDhX2zGDfPpg5E4YPj+7HfeWVNvPjjBk2n0yaMKR3fmoIuY9JC4sYM30VL0yawfbmXdi8eFPw9rVpYwV82zZo0eLw/vJyO9ahM08jRi12JyrsmcHMmdbajtQN46dfP2jUCN5919VmeQl/+OWuzdvosu0H5rboEjr8Mt8n9oHuGP8Atg6eRowKuxMV9sxg8mRo2BDOOCO682vVshOaJk/W70OU+MMvj9+4iiwMX+cfHTr80i/sGzZU3l9SYp/dGDz1GJGsoPSCiGwRkaWOfUeIyAci8o3vuUl8mpkgNAlY+mMMTJ8O/fvHJggXX2xztn/xhXtt8xD+MMuCDcs5JFksOrJrpf2VaNPGPgda7P4xjroxRvaoK6ZaxgODAvaNBGYaY7oAM33b6YsmAUt/1qyB9eutsMfCeedBdra6Y6LEH2Z50oZlLGvZib2161XaX4mWLe1vL1DY9/mWUY5C2CctLOKO1xcBcNvLX7Fhe/ovyRwJYQu7MeYTIDBh9WDgRd/rF4EhLrUrOagrJv2ZOdM+n3NObPU0aWJdOSrsUTFiYDcaZZXTe+MqvmzTE6gm/DInB448sqorxi/s9epFdG2/f3/bHuvK2banhCVFO9ldcqiGMzOHWH3sLY0xmwB8zy1CFRSR4SKyQEQWbN26NcbLxgkV9vRn5kz71z5UaF0kXHyxTSkbalakEpIhvfP5V1dDnbKDfNmmZ83hl/n5VS32/T63TYTC7vfvG8fg6aFyw097D0ZUTzqTsMFTY8xYY0yBMaagefPmibpsZKiwpzfl5fDRR9ZadyMiwh9VM3ly7HV5kDO3rATg2ed+y5yRZ1cfihkwSWnSwiJuG/spAHe8syqiZGaBfnzxuVfLDnnntx2rsG8WkSMBfM9bYm9SElFhT28WL4affordDeOnc2ebuErdMdExfbpdxKRFyD/yh8nPr3DF+F0pe3bsAmDDASLKVHnYj1/55p6T7Z0gwFh7+i5wo+/1jcA7MdaXXFTY0xu3/OtOLroIPv3UrpuaQqR07nWwC2t8+ilccEF45Vu3tufs3VvhSqlbZn3kB3JrR5SpMlh6hewsoWl974RNRhLu+BowD+gmIhtEZBgwGjhXRL4BzvVtpy8q7OnNzJk2hUDr1u7VedFFUFYG06a5V2eMpHzudbCfRWkpnH9+eOX9n9mmTRWulNpl1ie+P9dOUAo3U6U/vUKzBlbImzeoxTH5jWlYxzsT7SOJirnaGHOkMSbXGNPGGPO8MeYnY8w5xpguvuf0XuZdhT19OXgQPvnEXWsdoG9faNoUpkxJGSs55XOvA0ydanOvh8roGMiRR9rnjRsrXCl1S63Fvj/HCnskmSqH9M7nX9faNW6fvOYE2uTVCfvcTMA7TqdwUGFPXz7/3IbHuS3s2dlw/vkcnDyFeycsSgkrOaVzr4P9HN5+27phcnPDOmXmTus6uf2fU9lbUkZutlQI+4HcWrFlqtTsjh5HhT19mTnTfn79+rlf94UXUmtnMd3XLau0O1lWckrnXgd46y2bf/2WW8IqPmlhEXd//hMALfdsp3h/KRhoImUANGmWF12mSg8JeSDecTqFgwp7+jJzJhQU2FzqbjNwIKVZ2fT/9gsW+Cbb+EmGlTxiQFemP/w8Zy/7jA47NnIwJ4dl+d3Zc92NnDb6IzYW76d1Xl1GDOyWnGyPzz1nV60688ywio+ZvorN2XU5kFOLFnusN7e03NCw3CYBm3XvIE+LdDSosDtRYU8L/Olg/QI26vR8Lpw/H0aMiM8FGzdmccdjOefbLxjd76ZKhxJuJa9ezZBf38iQzz9nV50GLGvRkSaHShg2fyIy9y3y+lzEw2feQFGxDREE4ibugZ/DiIHdGLJlqc2//sQTYYtxUfF+EGFL/SYVwg5QumevnZwUq6h7ME2IumKcaBKwlCdYRMiUJ9+wkStu+9cd1B4ymC4//UC7HZsq9iV8haJp0+y/km++geefp9Gu7ZyyrpAeP6zkkjtf5eXe5zP0q3d589WRHLFvp/uuokOHbFK0KVOY/dIU7n9jQaXPYcxLn7LvppvhqKNsLvwwyfYJ9+YGTWnpEPZ6ZSWxJQDzsJWvFrsTtdhTnmARISeu+ZqSnFrUDrX0mgsc86vr4J/3c+nGhTze5MjEuzqmTrUrO/knTLVtW+nwkvJ6FA74FZ90PIEn3/07b71yJ9dd+Vc24sIs70OH4Omn4cEHYfNmAPoB87Nz+Cq/B/PaHUO2KefnS2aStX8nTPokohzqh3wW9eYGR3D01u8r9tcuLYk4nUBQ1GL3OFm+t8ODX4R0IZhP+/TvF/JFm56xp3etjs6doXt3fndgNWtHX1DzFHk3+fpruOIKOOYYmzIhQNQnLSwiy2edftjlZK678gGa7y3mpTfuo3tujPlRioth0CC44w7o2RNeew3mz+eXl9zN+D4X0/jAHv7w2SvcMed1ttVvzNVXPggnnRTRJfJ97qytDZrQ3GGxHyFlsQm702L32G9aLXYnTmH38N+4VKZ1Xl3rk/XRYvdPdNu2nvePP5efxfviF10Ejz1mZ0g2ahTvq1k2brTXbdbMLtDdpPKSB37X1CGHcC1o05ObL7uPl964j1fe+SuMHAD160d+7R074KyzYPlyGDcOhg2r+F0sPWkP04vtP6T6JTYL497a9SpEOhJGDOzGiLcWs7lBUxod3Ee9g/sprVOPXo2yYH+DyNsdiIY7ehy/sKs7JmUZMbAbuVmHf6Cnr7M5t2e1Oy7+MeUXXmhnU77/fnyv46e8HK6/3qYzmDwZWrWqUiSYawrgq3bHsGj0UxyxbJFdw9W/zFwIAidfTZ6z2s4aXbHCXvvmmysJo3Pa/t7a9dhbu15sYw5iXTGAHUAVyN29q8qNLLI6vSPkgaiwO1Fhj5l4z84c0jufBo6p4ad9v4ht9RpT2KxD/GPKTzvNJrSaMCG+1/Hz3HPW9fLII3DssUGLhAq3LDeGk/94Czz1lPXP33JLSHdE4ID0tm07aXbdFZR/+SW8/joMHFjlHP+0/fy8ugjUnJa3GsZMX0XpIcMWv7Dv3UHpIcOuTVviE77qAdQV40SFPSb8AuG3IP2zM8HdkLvifT7r0xhOX7eYue2Pw0hW/GPKs7Phsstg/HjYuzc690a4rF1rwzfPPbfaiT6BrinnfgBuvdUOeN5/v70pPfxwlbJOq79WWSlPTXqIU75fzP9dPpI/XXJJyGsP6Z3vyufq/9z8FnvL3XayUr19e1wR9jnfbKVk5RbabN7DTaM/Sl58fwJRi91JBgl7MvKaJCqHiV+0um5bR8s92/m0w/GV9seVyy+3C0C89178rlFeDkOH2hvJ889X61IIlsmwikvkT3+C226DMWPg7rurWO5+Ya1TeoBxEx+g/5ovuWfAbfyn0+nu9aka/J/bFqcrBmhcsjc2Yfe9by98traKsZFSCdPigAq7kwwR9mRl/0tUDhO/mP3se+tfn9Ph+MTFlJ9xhrV833orftd4+mmYPdu6YAIiYAIJyyUiYicMDR8ODz1kfe47dlQcbp1Xl/Y7NvL6a6P42dqFjDjvDl7pfX7CJl/5P89dtevb2ad7d9BIDlGntMQVi/1gWYonTIsD6opxkiHCXp3lHM+/oDW6BVzC34eWb/6Z747IR9q156FE/b3Ozrbx5P/9b3zcMWvWwF132RDDoUPDOiUsl0h2Njz7LHTqBPfeCzNmwDXXQIcOvP7xXFrOmMr+3NrcesndzOh6SkInX/nbPmb6KjY3OIKOJTv5+zntbBJwFyx2OLyKkp+USZgWJ1TYnWSIsCcr+9+Igd0q+dghfrMzh3RrAusL4ZZbmDPybNfrr5arrrIiOXGijVpxi/JyuOkmmxFx3DhXozoOT//vxZm/eoq/rpxCmxdfhH37aNu0KWuuuJ4/dBjIYtOA/CTkmam4OU09ivbZZdDWZwzEEhUTgHG8nSmTMC1OuCLsIvL/gJsBAywBbjLGHHCj7oSSIcKeKMs5EKflFfdEVB9+CAcOwODB7tddE2ecYScsvfCCu8L+r3/ZVYfGj7drgEZA0Lwtvvc9cFB7dr18zj35Nh4a/TRDOjeERo3oLMIk93oSPe3bw2ef2YlR4IorpnbAkngJTwWRBGL2sYtIPnAHUGCM6QVkA1fFWm9SyBBhD2tALU4M6Z3PnJFnx3V25qSFRUx+4Bl21a7PGZ8fSvxAmIi1rGfPtq4TN1i9GkaNsrHyN9wQ0ak1jamEdM3NWG0Xw0ileO/OneGHH+DHH+22C66Yoad1oG4t+3uIJSwznXBr8DQHqCsiOUA9YKNL9SaWDBF2N2OMY6K8HObOhZdfhgULXJnWPWlhEfdMWMQpy+cxu1Mf1u8uTU6Uwy9+Yb8v//lP7HUdOmRvFHXq2Nj1CIW2pmiklF+Yw8lRR9nvzaxZdrtdu5irPLVzU87q1oJuLRsmNhVEEonZFWOMKRKRfwDrgf3ADGPMjMByIjIcGA7QzoUPKy74f1BpLuzgXoxxJDjdAb3Zxdjpj9Fs0ReHC/Tvb33TnTtHfY0x01fRbf0Kmu3byQdHnQwkZmC4Cvn5doBz/Hj485/DXikoKI89dvgGGMV6rTUJd7Jcc1Hh/27MmGETicWyfm0q/RNJMG64YpoAg4GOQGugvohcF1jOGDPWGFNgjClo3tyFjHPxQJOARY3THdBy1zaefup26ixfwsJ7Rttp6Y8+Cl9+CaeeCosXR32djcX7Ofeb+ZRmZfNxpz6V9iecX/0KiopiC31csgTuuQeGDLFRKlFQ04pKyXTNRUz37vZ5xQro2PHwbzJWPPabdmPwtD+w1hizFUBEJgKnAi+7UHdiyRBXTDLwuwNqlZUybuID1D+4n59f+zC7G/RgTvfu9gc7aJCdSTlggM3r3b59xNdp3bgOF678lLntj2NXncMJohJlfTr/leQ3qsN7Hbtg/vIg569rycadByIbMD5wAK691vqRx46N2sKsKRopoYPasXLEETY18fLlMf2zq4QHk4C5Iezrgb4iUg/rijkHWOBCvYlHhT1q/BbzL+dP4JjNa7jl0ntZ2aIj4rSku3eHDz6Avn3h4ovtAtQRptp9qOUu2u7czCOnX1uxL1HWZ2B0yYZdJfytxwWMnvoY3b/6mKKjTo4sjcLdd1uLfepUiOFfbDjCnQzXXNQMGGCFfcCA2OrxkJAH4oaPfb6ITAC+BsqAhcDYWOtNCirsUdM6ry5Z36/l9nlvMqX7z/igS9+K/ZXo3h1efdWuYD9yJDz+eETXOeOL6ZTVqcv8484An4FaOycxE6iDDVJOOLoft8x7i7tmv8jsTgUcysoOz+c/ZYp1T/361zaLYoyklXDXxF/+YlM3uLVwisfcMOBSVIwx5s/GmO7GmF7GmOuNMSVu1JtwVNijZsTAbvx+3usYER44exhQ1ZKuyF/zieHNUy+x09ynTau2XmfOm/5/mUrpK6+x6ayB7Mg6vEJP8f7ERMYE8+OXZefw8Jk30vWn9Vy78L1qy1awbBlcfTWccELQpFyep1Ejd0Tdwxa75opxosIeNUMa7mfwsllMOvkitjRsViXEMjDW+r5Trmd18w4cuO4G2Lo1aJ2B5/SdM5XcPbu4r/UZCUk2FkgoP/70LqfwcccTGPnxeDpsL6q2LJs2WTdUgwbwzjvuLP2mVI9a7B5HhT16HnmErNxcrprwZNDJSYFujJKcWvzmoj8iO4ttDHeQH5/zHDHl3PTVZBa36sLspl2CNiERKRMCo0tys4TcnCzuPO8ODmbn8tz/HqRV2b7gPv/166FfP5tGd9KkiGeXKhGiFrsCqLBHy969Ngb78suDrvIDwUV3VfMOPNRvqB08fPLJas+5aMUndN6+gedPHBLyB5uIlAmBE7/GXH4cY35+HDlt23LbkFF03LGJaVP+wpAG+w6faIwV8oICa7FPmwYnnxzXtioBeMxq1yRgTlTYo+PNN2H37qgWhPjg7Mu5v04R/PGPNlrmxBOrnFO7tIQ7P36RJS07M/non5FXN5eSsvKEJBsLJNQgpd13NtzQm1rXXgu9etnwzpYtYd48WLrU7pswAbqlYPx4JuPBcEe12J2osEfHuHE22uX00AszhJwkM6i7nZafn28XbV67tso59856nja7tvK3s4ZRp1Yu91/osuTtAAAXnUlEQVTcMzVSJgTjvPOgsNBOXlq61FrqTZvaVAELF6qoJxIPCXkgarE7UWGPnGXLrEX6z39W+0OqMdZ66lS7puipp1qr9rTTGHJ8a4566Tl6LXyPZ0++jPXHnVwp73pKCHkw2rSxaQIeeyzZLVHAc24YUGGvjAp75IwbB7VqhZWRsNpY66OPtulazz/fWv6nnAK7d9Nr6VK49FJuffMNbs3ODn6uogTDwxa7umKcZFASsIRw4AC89BJccgk0axZ7fT162JmYf/ub3W7Rwt443nzTrgCkKEpYqMXuRJOARcbEibB9O3c0KGDyyKnu5CBp2NDmJR81yr12Kt7Gg79nFXYn6oqJiG2P/It9ea2Y3LRbpQUeIIX934p3cLpiPCbu6opxkinCboy1pm+8EYYNg/fec/+LvXo1zb76nNePHYCRw18jL6wAr6QZGu7ocTJB2Pfvt+twXnYZTJ9up61fcIEV+NJS967z739TJlm8dUz/KodScmUexXt4SMgDUWF3ku7CbgwMHWqzJz7wgF0EYtMmuO8+Gyt+7bXuWO4HD8L48czpcSpbGxxR5XBKrsyjKB5Chd1Jugv7E0/A66/bqJJ777WRJLm58H//B6NH25V+/v732K/z7ruwdSu5tw5Pn5V5FO/iMf86qLBXJp2Ffd06u3DDBRfAXXdVPX7nnXDFFdZ6X7IktmuNGwft2nHqr65J3RmgiuJhV4wrUTEikgf8G+gFGGCoMWaeG3UnlHQW9t/9zj4//XTwL7QIPPUUfPSRzekyZ050seFr1thVkP78Z8jOzqwFHpTMRC32qHkcmGaM6Q4cB6xwqd7Ekq7CPn++zUly993Qrl3ocs2a2Wnu8+dbkY+GZ56xN4RqEn4pSkqg4Y7RIyKNgDOA5wGMMQeNMcWx1psU0lXY//QnaN6cyf0ur1ht6LTRHwVfUeiaa+xakvfdZwdWI2HfPnjhBTvTtHVrd9quKPFGwx2johOwFfiPiCwUkX+LSP3AQiIyXEQWiMiCrSFWzEk66Sjsn34KM2aw9LpbuXPadxWrDfknC1URdxH4179sWOSdd0Z2rddfhx077DqdipLqeEjIA3FD2HOAE4BnjDG9gb3AyMBCxpixxpgCY0xB8xhWZI8rKSrsznU/K1nixljLu1Ur7sjrG/5ycV27wogRdnGMTz4JrxHG2Kibnj3hjDNi7JGiKPHEDWHfAGwwxsz3bU/ACn36kYJJwALX/axkiX/0EXz8Mdx9N2v3BfchFhXvD+6a8fvjb78dyspqbsiUKbB4sV0Qw8OWkJKGeMy/Di4IuzHmR+AHEfEHL58DLI+13qSQ6CRgL7wAnTrBUUfZwcwa1v30s7/0EGOmrbTWeps2cMst1U4KCuqaqV/fDqQuWVLzQKoxNha+Uyc7yUlR0gEPGyBuRcX8BnhFRAqB44G/uVRvYkmkK+bVV+00/1at7OpBt98OI0dWEfdQ0/N7LPjYLnBx331Qp07QFYoCqeKaGTIEBg60dWzcGPrEl1+GBQvspKfc3LC7qCgpgQctdlfi2I0xi4ACN+pKKnEW9kkLixgzfRW7Nv/EZ8/dStmxfWj68ceQk2MHJB9+2OYg/8MfKs4JtlZoVvkhRs15yfrKhw4Fqq5QFOqrXOlG4R9IPe44a4l/+GHV2Paffjq8HumNN8b8HihKwtBwRwU4LOyHDlVfLgqcvvKrF71P4/27ubXgeiYt3WK/gE8+CT//uY1UmTat4rxglviVKz+m0+bv4cEH7U3Bx5De+cwZeTZrR19AfgjXTBWXTZcuNjZ99mz4zW8q39RKS+1s1R07bJks/booaYyHXDP6S3Xit1bjIOwVvnJjuGrxdOa36cmXzY867BrJyoLx4+1K9lddBatXA1asndP2j849yJ/nvgQnncSkTn1Dxq2HXDw6WB6XG2+0UTLPPAPXXWcXlP7mG7j0UjtA++9/w/HHu/6eKEpC8Ji1DrrQRmX81m8chN3vAjn2x2/otGMjz/S9vNJ+wA5ovvMOnHgiDB7MlOcm8tDcTRWLPz96xXEM+eddsHMHH/3+FUb9b2nFwGrgIhc1Lh4dyN//Dnl5drLTa6/ZfbVr24HVMNYzVZSUw0MWeiAq7E78Fns44X8R4veVn7VmAeUIHx51UsX+SnToABMmUN6/P42G3sBPg0dicutQVLyfDb8bCZ/alLz3rc1mf+nBSqf6B0f94h1RHhcRGwJ51VXw/vv2H8SFF0LbtrF2XVGSi1rsHidOrphJC4vYd9DeLM5c+xWFR3ZhR73GoV0jZ57JPy68nTsnPcaU8b/jrWP7c/zG1Zy3ei5T+gziwnvuYeOo94JeK+ZFLjp10pmlSmbgYYtdfexO4iDs/kHTHftKydu/i+M3rmZ2pz7k1c2tNsXtM936c82Vf0UwjJo9ntPWLeafp1/L787+FYiEjFvXRS4UJQC12D1OHITdOcHoZ2sXkoXh4459KN5fWjFwGkzcW+fVZW6H4znn5mdpVLKXPbXqUp6VXRHtMmJgN0ZNXFJp8pIucqEoDjTcUQHiIuxO10ifohXsza1D4ZFdgGoSdeGIahFhV50GlGdlVxLuwGgZXeRCUWrAQ64Ztdid+KNiXBw8dU4w6r1xFYVHduFQ1uEwxMABTz/hRLXoIheKEgYes9ZBhb0ycbDY/S6T8n376LHlO8addEmVMqEGPFW4FSUGPGShB6LC7iQOwu4X5mnPvU1u+SEWtu5epYwOeCpKHFGL3ePEKdxxSO98hnS17p2V7Y6udEwHPBUlTnjYYtfBUydxTCnAl19Cu3b84YYzdcBTURKNx6x2tdidxDGlAIWFcPzx6jdXlESja556nHilFDhwAFatgmOPdbdeRVFC4yEhD8Q1YReRbN9i1lPcqjPhxCtt74oVts5jjnG3XkVRasZjbhhw12L/LbDCxfqSQ3a2+8JeWGif1WJXlMThYYvdFR+7iLQBLgAeBH7vRp1JI4Sw+1c/CisFbiCFhVCnjl3bVFEUJc64NXj6GHAn0DBUAREZDgwHaNeunUuXjQNBhN2fyCtU7vMaWbIEevastNqRoigJQl0xkSMiFwJbjDFfVVfOGDPWGFNgjClo3rx5rJeNHzk5VQZPnYm8/FRZGLo6CgvVDaMoiUaTgMXEacDFIvI98Dpwtoi87EK9ySGIxR5qyn9Yuc83b7YPHThVlOSg4Y6RY4wZZYxpY4zpAFwFfGSMuS7mliWLIMIeU+7zJdZloxa7oiQYDwl5IBrHHkgQYY9oYehAVNgVJbl4zA0DLs88NcbMBma7WWfCCSLszhS6pRuKqHNkK35/fo/wBk4LC6FVK0jlcQVFyUQ8bLFrmEYgOTlBwx2H9M5nyLblMOoGOOUUGPVZePUVFqp/XVGUhKKumECys0OnFHjkEfs8bx68/37NdZWVwbJl6oZRlGTiQVeMCnsgoWaelpXB7Nlw++1wxBEwYULNdX3zDZSUqMWuKMnAw+GO6ooJJJSwr1xpk3n17QtbtsCMGfbLUp0fz59K4Ljj4tNWRVFqRsMdlZDCvnChfe7dG/r3h40brUVeHYWFtr6jj66+nKIo7uMhIQ9ELfZAQgn7ihV2YLVr18PHv/zSboeisBC6d4fatePTVkVRlCCoxR5IiKgY1q2Dtm3t8R49oH59+OKL6uvSVAKKknw85l8HFfaqhIqKWbcO/MnLsrOhT5/qhb24GNavV/+6oiQLD7tiVNgDCeWKWb8e2rc/vH3SSdbvfvBg8Hp0xqmipAZqsStBhb20FIqKqgp7SclhAQ9EF9dQlOTi4XBHFfZAggn7jz9CeTm0aQPY/OyXLSgFYMxfX2bSwqKq9RQW2nj31q3j3WJFUcLBQ64ZjYoJJJiwb9lin1u2PLzoBo3YVq8x7dcsDb7oxuLF1lr30JdJUVISj1nroBZ7VYIstFEh7C1aHF50Q4TFR3bluE2rqy66UVpqhf344xPXbkVRKuNho0qFPZDqLPYWLSotrlHYqgtdtv1A/ZJ9lRfdWLTIzlI99dQENFhRlGpRi12pSdidi2ssPrIrWRh6bV5TedGNuXPt8ymnxLmxiqKERC326BGRtiIyS0RWiMgyEfmtGw1LGgHCPmlhEa9OXsCBnFqc9uQXnNW9ecWiG4uP7ALAiZu/qbzoxty5NubdN9iqKEoSUYs9KsqAPxhjjgb6Ar8WkR4u1JscHMLuHyitvWMb2+o1pmjnAd7+qojL+uSTn1eX4nqNWd+sDdcdXFd54HTuXHXDKEqy8XC4Y8xRMcaYTcAm3+vdIrICyAeWx1p3UnCkFPAPlB6xbyc76jYCYH/pIWat3MqckWfb8sUXwKuv2gHXnBxYuxY2bFBhV5RUw0OuGVd97CLSAegNzA9ybLiILBCRBVu3bnXzsu7iSCngHxBtdGAvxXUaVhSpNFDavz/s3g1z5tjtyZPt86BBCWmuoig14DFrHVwUdhFpALwN/M4YsyvwuDFmrDGmwBhT0DyV1/90uGL8A6KND+xhV536FUUqDZQOGgR16hxeeOOtt2xGxy5dEtZkRVGC4CELPRBXhF1EcrGi/ooxZqIbdSYNh7CPGNiNurnZNC7Zw846DQCom5tdeaC0QQMYMgRefBGmTYPPPoObb05GyxVFCYYHLfaYfewiIsDzwApjzCOxNynJOITdPyDaaPRedtVpQH5eXUYM7FZ5oBTg7rutpX7eeTaFwLBhiW61oiiBeNhidyOlwGnA9cASEVnk23e3MeY9F+pOPAHhjkO6HwFlpdw6uA+3+gdMAznmGJg6Ff73P7jtNsjLS1BjFUUJC49Z7W5ExXwGZM6tMTClQHGxfW7SpPrzBg60D0VRUgtd81SpMvN0xw77rFa4oqQXHhLyQFTYA4nWYlcUJTXxmBsGVNirUquWzc7oRy12RUlP1GJXKqhVq/Jyd35hV4tdUZQ0QYU9EL/F7v/75nfFqMWuKOmJumIUatWyz353jLpiFCU98XASMBX2QPzC7nfHFBdDvXqH9yuKkl5ouKNSRdh37FD/uqKkIx4S8kBU2AMJZrGrG0ZR0hePuWFAhb0qarErSmagFrtSQTBhV4tdUZQ0QoU9kGCuGLXYFSV9UVeMQm6ufVYfu6KkNxruqFTgtNjLy2HXLhV2RUlnNNxRqSTsu3bZL4UKu6KkHx4S8kDcWhpvkIisEpFvRWSkG3UmDaewazoBRVHSkJiFXUSygaeA84AewNUi0iPWepOGCruiZBYe86+DO0vjnQR8a4z5DkBEXgcGA8tdqDvhzPqumLOAW8bNpVbTJjwFKuyKko6oKyYm8oEfHNsbfPsqISLDRWSBiCzYunWrC5d1n0kLi3j0k+8ByDlUysGftgMw68eD1ZylKEpK40GL3Q1hD3ZbrPJOGmPGGmMKjDEFzZs3d+Gy7jNm+ir2lNu3JLf8EI0O7AXg6UXbktksRVGiQcMdY2ID0Nax3QbY6EK9CWdj8X4OZlvvVO6hMhqV7AHg25LsZDZLUZRY0HDHqPgS6CIiHUWkFnAV8K4L9Sac1nl1Ka0Q9tIKi71hi2bJbJaiKNHgISEPJGZhN8aUAbcD04EVwJvGmGWx1psMRgzsRnbt2gDklpfRsGQve2vV5ffnp2+Qj6Io3sONqBiMMe8B77lRVzIZ0jufnN3HwaNQ61AZrUwJ0iSPIb2rjAUripIueMy/Di4JeyZx4YkdALhvwFEw/ycoaZrcBimKEh3qilEqCJygpDHsipLeeNBiV2EPJDsbsrIOC3vjxslukaIo0aDhjkolateGAwfUYleUTMJDrhkV9mA0amQzO+7cqcKuKOmOx6x1UGEPTuPG1lpXi11R0hcPWeiBqLAHo3FjKCqyC22osCtKeqMWuwJYYV+37vBrRVHSD7XYlUo0bgwbNtjXKZqwTFGUCPCY1a7CHgynld66dfLaoShK7GgSMAVQYVeUTMBDQh6ICnswnMLesmXy2qEoSux4zA0DKuzB8Qt7bq59KIqSfqjFrlTCL+wa6qgo6Y8HLXbN7hiMAQPgiiugX79kt0RRlGjxsMUek7CLyBjgIuAgsAa4yRhT7EbDkkp+PrzxRrJboSiKW3jMao/VFfMB0MsYcyywGhgVe5MURVFcRMMdI8MYM8O3NB7A59iFrBVFUZKPh4Q8EDcHT4cC77tYn6IoSux4zA0DYfjYReRDoFWQQ/cYY97xlbkHKANeqaae4cBwgHbt2kXVWEVRlLDxsMVeo7AbY/pXd1xEbgQuBM4xJvSt0RgzFhgLUFBQ4L1bqKIoSoKINSpmEHAXcKYxZp87TVIURXERD7piYvWxPwk0BD4QkUUi8qwLbVIURYkdD695GpPFbow5yq2GKIqixAUNd1QURckQPCTkgaiwK4qiZBgq7IqiZDYe86+DCruiKJmKumIURVEyFLXYFUVRMgQPhzuqsCuKktlouKOiKEqG4CEhD0SFXVEUJcNQYVcUJbPxmH8dVNgVRclU1BWjKIqSoajFriiKkoF4TNxV2BVF8QYecs2osCuKktl4zFoHFXZFUTIZD1npTlwRdhH5o4gYEWnmRn2KoiiuoRZ75IhIW+BcYH3szVEURXERtdij5lHgTsB7t0VFUVIfD1rsMa15KiIXA0XGmMVSw51RRIYDw32be0RkVSzXTgDNgG3JboQLZEo/QPuSiqR+Px588PDr6nUq9fsC7cMpJKaGu5mIfAi0CnLoHuBuYIAxZqeIfA8UGGNS/Y0JCxFZYIwpSHY7YiVT+gHal1QkU/oBmdWXGi12Y0z/YPtF5BigI+C31tsAX4vIScaYH11tpaIoihI2UbtijDFLgBb+7Uyz2BVFUdIVjWMPzdhkN8AlMqUfoH1JRTKlH5BBfanRx64oiqKkF2qxK4qiZBgq7IqiKBmGp4VdRAaJyCoR+VZERgY5XltE3vAdny8iHRLfyvAIoy+/F5HlIlIoIjNFJKx42GRQU18c5X7uS2WRkiFq4fRDRK7wfS7LROTVRLcxXML4frUTkVkistD3HTs/Ge2sCRF5QUS2iMjSEMdFRJ7w9bNQRE5IdBtdwRjjyQeQDawBOgG1gMVAj4AytwHP+l5fBbyR7HbH0JezgHq+179K5774yjUEPgE+x0ZjJb3tUXwmXYCFQBPfdotktzuGvowFfuV73QP4PtntDtGXM4ATgKUhjp8PvA8I0BeYn+w2R/PwssV+EvCtMeY7Y8xB4HVgcECZwcCLvtcTgHOkpim2yaHGvhhjZhlj9vk2P8fOO0hFwvlcAB4AHgYOJLJxERBOP24BnjLG7AAwxmxJcBvDJZy+GKCR73VjYGMC2xc2xphPgO3VFBkM/NdYPgfyROTIxLTOPbws7PnAD47tDb59QcsYY8qAnUDThLQuMsLpi5NhWKskFamxLyLSG2hrjJmSyIZFSDifSVegq4jMEZHPRWRQwloXGeH05X7gOhHZALwH/CYxTXOdSH9LKUlMuWLSnGCWd2DsZzhlUoGw2yki1wEFwJlxbVH0VNsXEcnCJp77RaIaFCXhfCY5WHdMP+w/qE9FpJcxpjjObYuUcPpyNTDeGPNPETkFeMnXl/L4N89V0uU3Xy1ettg3AG0d222o+vexooyI5GD/Ylb3Ny5ZhNMXRKQ/NsfPxcaYkgS1LVJq6ktDoBcw2zfbuS/wbgoOoIb7/XrHGFNqjFkLrMIKfaoRTl+GAW8CGGPmAXWwSbXSjbB+S6mOl4X9S6CLiHQUkVrYwdF3A8q8C9zoe/1z4CPjG2FJMWrsi8998RxW1FPVlws19MUYs9MY08wY08EY0wE7XnCxMWZBcpobknC+X5Owg9r4FqnpCnyX0FaGRzh9WQ+cAyAiR2OFfWtCW+kO7wI3+KJj+gI7jTGbkt2oiEn26G0yH9gR8NXYEf97fPv+DysUYL+cbwHfAl8AnZLd5hj68iGwGVjke7yb7DZH25eAsrNJwaiYMD8TAR4BlgNLgKuS3eYY+tIDmIONmFmEzfqa9HYH6cdrwCagFGudDwNuBW51fCZP+fq5JFW/WzU9NKWAoihKhuFlV4yiKEpGosKuKIqSYaiwK4qiZBgq7IqiKBmGCruiKEqGocKuKIqSYaiwK4qiZBj/Hym73EmYvin8AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f458036aef0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_poly = PolynomialFeatures(degree=30).fit_transform(X)\n",
    "X_test_poly = PolynomialFeatures(degree=30).fit_transform(X_test)\n",
    "\n",
    "model = LinearRegression()\n",
    "model.fit(X_poly, y)\n",
    "y_test = model.predict(X_test_poly)\n",
    "\n",
    "plt.scatter(X.ravel(), y)\n",
    "plt.plot(X_test.ravel(), y_test,c='r')\n",
    "plt.title(\"mean squared error: {0:.3g}\".format(mean_squared_error(model.predict(X_poly), y)))\n",
    "plt.ylim(-4, 14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we increase the degree to this extent, it's clear that the resulting fit is no longer reflecting the true underlying distribution, but is more sensitive to the noise in the training data. For this reason, we call it a **high-variance model**, and we say that it **over-fits** the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just for fun, let's use IPython's interact capability (only in IPython 2.0+) to explore this interactively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEICAYAAABLdt/UAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAGStJREFUeJzt3XucXGWd5/HPt7tzISEkMDS3QAzITWTmNWi/FHUXUECDi+C67gygIyhrBp0V1/XCbS7OrLPrjs4wzsLKZEYGEQYviBgdkKu8GBzD2oFwB0VAkhBNIxe5JunOb/84TzWnK1XddTnp7jz5vl+veqXqPOc853mqK996znNOVSkiMDOzfPRMdQPMzKxaDnYzs8w42M3MMuNgNzPLjIPdzCwzDnYzs8w42G27JCkk7T/V7TDbGhzsZtsASe+T9Hzp9mJ6c3p9k/UXS7pG0tOSfinpAkl9pfJ3Sbo31fVvkg7pom1/JWm1pN9I+oWk8+rKl0l6SNJmSad1uh9rnYPdtnmSeqdw332tLGu3jnoRcXlE7Fi7AR8FHgHuaLLJ/wXWA3sCvwscmbZB0gHA5cAZwALge8Dydttd8hXg4IjYCXgzcIqk95TK70r7btZWq5iDfRsh6TFJn5Z0t6QXJH1F0u6SrpX0nKQbJe1cWv/wNBJ7RtJdko4qlX1Q0gNpu0ck/WGp7ChJayR9UtJ6SeskfXCcdp2W6nhO0qOS3peW90r6oqQnU/kfpRFmX6k/x5Tq+ayky0qPv5VGms9KulXSa0tll0j6chqRvgC8VdKstL/HJf1K0kWSdiht8+nUlyckfWiC53p+en7XSVor6XO1N4/U3x9JOl/SU8BnmyzrkfTHaQS7XtKlkuanOhan5+J0SY8DN4/XniZOBS6N5h8d3xf4ZkS8HBG/BH4A1J7DdwD/GhG3RcQw8L+BhRThT2r72ZJ+LunXkr4paZdmDYmIhyLihdKizcD+pfILI+Im4OUO+mkdcLBvW/4TcCxwIPAu4FrgXGBXir/lmQCSFgL/AnwO2AX4FPBtSf2pnvXA8cBOwAeB8yW9rrSfPYD5FP/ZTwcuLL9p1EiaC/wdcFxEzKMYra1KxR9O+zgMGADe22ZfrwUOAHajGOldXld+CvCXwDzgNopwOpBidLp/avufpnYuSc/BsanOYxjfV4HhVM9hwNuB/1IqfyPFaHm31IZGy05Lt7cC+wE7AhfU7edI4DUUQUt60z5lgrYh6VXAEcCl46z2JeAkSXPS6+E4inAHULpR9/jQ9PhM4N2pfXsBTwMXTtCmsyU9D6wB5gL/PFE/bCuKCN+2gRvwGPC+0uNvA18uPf4YcHW6fxbwtbrtrwNObVL31cDH0/2jgJeAvlL5euDwBtvNBZ6heMPZoa7sZuCM0uO3A1GrN/XnmFL5Z4HLmrRvQdp2fnp8CcVotVYu4AXg1aVlbwIeTfcvBj5fKjsw1bd/g33tDmwo9wc4Gfhhun8a8HjdNo2W3QR8tPT4IGAT0AcsTvvfr8PXwp8At0ywzmuAlRRvUJGeM6Wyg9PzdRQwM9W3GTgnlT8AHF2qa89a2yfYpyjeCP8cmNeg/DbgtKn+v7Q93Dxi37b8qnT/pQaPd0z3XwX85zQN84ykZ4B/R/EfFEnHSVoh6alU9k6KUX/Nr6M4RK95sVT3qCgOv3+fYq52naR/kXRwKt4LWF1a/RetdjJN43w+TQX8huJNgLo2luvuB+YAK0v9/UFa3m5bXgXMSP2p1fX3FCPxRvtutmyvuv38giLUd5+gnlZ8gOKooiFJPRRv5FdRvPnuCuxMcVRDRDxIMZVzAbAuld9PMdqG4jn4Tqn/DwAjwO5piqt2Avfc8n6jcCfFa/HPO+ybVcDBnqfVFCP2BaXb3Ij4vKRZFKP9LwK7R8QC4BrGHpq3LCKui4hjKd40HgT+IRWtA/YprbqobtMXKMK4Zo/S/VOAEymmTOZTjHCpa2N5bvlJijB5bam/86M4ydhKW8pWU4zYdy3VtVNEvLa0TqN57fplT1AEZHmfw4x9M277q1UlvYXiTePKcVbbhaK/F0TEhoj4NfBPFG/gxY4jroyIQyPit4A/S239SSpeTTG9Vn79zI6ItRFxRrxyEvd/Ntl/H/Dqdvtm1XGw5+ky4F2S3pFGv7PTSdG9KQ69ZwFDwLCk4yimSdqm4uTtCWmufQPwPMXIDuCbwJmS9k7z82fXbb6KYg54hqT6Ofh5qb5fU4R/swABICI2U7yhnC9pt9S2hZLeUWrLaZIOkTSHIsia1bUOuB74a0k7pROJr5Z05ARPR70rgE9I2lfSjqkP36g7EurEqcC3I+K5ZitExJPAo8BHJPVJWpC2u6u2jqTXp9dGP8URyffSSB7gIuAv01w+kvolndhoX+n5+UNJO6vwBuCPKKaiauvMlDSb4o15Rno9Onu2Ij+5GYqI1RQj3nMpAnw18GmgJwXCmRRh9zTF6Hh5h7vqAT5JMTp9itIldRRBex1FmNxBMS1Q9icUo7qnKQ7byyfbLqWYulhLMUWwooW2nAU8DKxI0zc3UsxrExHXAn9LMe//MBNfhfIBijfA+1P7riRNY7XhYuBrwK0UIfsyxXmQpiTdp3RVUZPy2cDv0WAaRtK5kq4tLXoPsITi7/8wxdHCJ0rlX6I4P/JQ+vfDdWXLgeslPUfx/L9xnKb/R+DnwHMUg4r/k24111McUb0ZWJbuHzFOfdal2skUs61K0mKKgJtRwajVzMbhEbuZWWZaDnZJF6cPWtzboOxT6QMXuzba1szMJk87I/ZLKObsxpC0D8UHPx6vqE2WoYh4LCLkaRizra/lYI+IWylOkNU7H/gMHVy6ZWZm1ev0S38AkHQCsDYi7pLGvwxa0lJgKcDcuXNff/DBB4+7vpmZjbVy5conI6J/ovU6DvZ0PfB5tHgNdEQso7jUiYGBgRgcHOx012Zm2yVJLX2Cu5urYl5N8Q1yd0l6DNgbuEPSHuNuZWZmW1XHI/aIuIfS92ekcB9In3ozM7Mp0s7ljlcAPwYOUvF93advvWaZmVmnWh6xR8TJE5Qv7ro1ZmbWNX/y1MwsMw52M7PMONjNzDLjYDczy4yD3cwsMw52M7PMONjNzDLjYDczy4yD3cwsMw52M7PMONjNzDLjYDczy4yD3cwsMw52M7PMONjNzDLTzg9tXCxpvaR7S8u+IOlBSXdL+o6kBVunmWZm1qp2RuyXAEvqlt0AHBoRvwP8FDinonaZmVmHWg72iLgVeKpu2fURMZwerqD4QWszM5tCVc6xfwi4tsL6zMysA5UEu6TzgGHg8nHWWSppUNLg0NBQFbs1M7MGug52SacCxwPvi4hotl5ELIuIgYgY6O/v73a3ZmbWRF83G0taApwFHBkRL1bTJDMz60Y7lzteAfwYOEjSGkmnAxcA84AbJK2SdNFWaqeZmbWo5RF7RJzcYPFXKmyLmZlVwJ88NTPLjIPdzCwzDnYzs8w42M3MMuNgNzPLjIPdzCwzDnYzs8w42M3MMuNgNzPLjIPdzCwzDnYzs8w42M3MMuNgNzPLjIPdzCwzDnYzs8w42M3MMtPOLyhdLGm9pHtLy3aRdIOkn6V/d946zTQzs1a1M2K/BFhSt+xs4KaIOAC4KT02M7Mp1HKwR8StwFN1i08EvprufxV4d0XtMjOzDnU7x757RKwDSP/u1mxFSUslDUoaHBoa6nK3ZmbWzKSdPI2IZRExEBED/f39k7VbM7PtTrfB/itJewKkf9d33yQzM+tGt8G+HDg13T8V+G6X9ZmZWZfaudzxCuDHwEGS1kg6Hfg8cKyknwHHpsdmZjaF+lpdMSJOblJ0dEVtMTOzCviTp2ZmmXGwm5llxsFuZpYZB7uZWWYc7GZmmXGwm5llxsFuZpYZB7uZWWYc7GZmmXGwm5llxsFuZpYZB7uZWWYc7GZmmXGwm5llxsFuZpaZSoJd0ick3SfpXklXSJpdRb1mZta+roNd0kLgTGAgIg4FeoGTuq3XzMw6U9VUTB+wg6Q+YA7wREX1mplZm1r+abxmImKtpC8CjwMvAddHxPX160laCiwFWLRoUbe7NbM2XH3nWr5w3UM88cxL7LVgBz79joN492ELp7pZW11EsGkk2DiymY3DxW3TyGY2pPvl5RtHRtg4PHbdjcMjbBzZzKaReGWb0XVr9aWykbR+ur+pVNeGUl3/8IEB/v0B/Vu1310Hu6SdgROBfYFngG9Jen9EXFZeLyKWAcsABgYGotv9mllrrr5zLedcdQ8vbRoBYO0zL3HOVfcAVBrumzcXQbYhhefGuvDcUArW+lDdMLKZTVsE7SvbNapvzP1xllVpZl8Ps3p7mNlX3GbU7peWzZ3Vx869pbK+V9aZ1dfDnvN3qLRNjXQd7MAxwKMRMQQg6SrgzcBl425l1qHtdfRZExEMb44xYbehLtxqZRtGNvMX37t/NNRrXto0wp9+914effKF0e021QVtfV0bG+yvvM3w5urGa709YmZvDzN6xcy+Xmb1bRmgM3rFTjNnjAZmrXxGn5jZ2zu63qzSdmPCtrxdray3tM1ofT2jbZFUWR+3piqC/XHgcElzKKZijgYGK6jXbAuTNfqsiYjGo8FW7o85pE+PGy0bd3rglZFsuTwqyNDfvDzMl276WRGeDUaXM0rBN2tGD/Nm96XyXmb0akxgFtv0plAdG5i1kC1Cuha0vWMCuhgJv7Kst2fbCNDpqoo59tslXQncAQwDd5KmXMzaMbI5SiPGkdH5y/K85ue+33j0+WfL7+PJ5zeU5jZHxoTjhtG6RrYIz9FD/Qahu2mkulGoxJgRYaPD+Jm9Pew4q4+Zc8Yu2+J+gyButu7SS1cy9PyGLdqz1/zZ3HbW2+hxiGZHUcVbf5sGBgZicNCD+qlSf0Jp9LC9wSiy0WH+xuGR0e3rTyhtccKodEJpNKTrTijV6hqp8FC+r0ctH3aPHrI3mRcdN4hr2zUI1THb9PXQ1zM1h/L1RzkAO8zo5X+957e3qymsHEhaGREDE61XxVSMjaN2QqnZIXjT5S2eGGpp3bo518pPKDUYJdbmRmvBN2dm3xZzllvMZTYIzPqTVJ/8xiqefGHjFm3YY6fZXPeJI0bD1Ifyr6iF9/Z8XmJ7k02wR0RxKD/BCZ6JTgxNdOa+/iTTRKFb5QmlHtHS4fi82X1NTjZtGZhFee/Y+c8Gc6D186Kj4d3bM6mj0D8+/pCGo8+zjzuY+TvMmLR2bGvefdhCB/l2ZJsK9gtu/hnL73pii7nTKk8o1fT1qKU5zh1n9407v9nKHGir6/b1+qt9PPo0m9g2Fey7zJ3Ffrvu2PDa0Pr51PKlTo3mUxtuV9u2t8cnlKYxjz7NxrdNBfspb1zEKW/0p1bNzMbjY3szs8w42M3MMuNgNzPLjIPdzCwzDnYzs8w42M3MMuNgNzPLjIPdzCwzDnYzs8w42M3MMlNJsEtaIOlKSQ9KekDSm6qo18zM2lfVd8V8CfhBRLxX0kxgTkX1mplZm7oOdkk7AUcApwFExEZgy19CMDOzSVHFVMx+wBDwT5LulPSPkubWryRpqaRBSYNDQ0MV7NbMzBqpItj7gNcBX46Iw4AXgLPrV4qIZRExEBED/f39FezWzMwaqSLY1wBrIuL29PhKiqA3M7Mp0HWwR8QvgdWSDkqLjgbu77ZeMzPrTFVXxXwMuDxdEfMI8MGK6jUzszZVEuwRsQoYqKIuMzPrjj95amaWGQe7mVlmHOxmZplxsJuZZcbBbmaWGQe7mVlmHOxmZplxsJuZZcbBbmaWGQe7mVlmHOxmZplxsJuZZcbBbmaWGQe7mVlmHOxmZpmpLNgl9aYfs/5+VXWamVn7qhyxfxx4oML6zMysA5UEu6S9gf8A/GMV9ZmZWeeqGrH/LfAZYHOzFSQtlTQoaXBoaKii3ZqZWb2ug13S8cD6iFg53noRsSwiBiJioL+/v9vdmplZE1WM2N8CnCDpMeDrwNskXVZBvWZm1oGugz0izomIvSNiMXAScHNEvL/rlpmZWUd8HbuZWWb6qqwsIm4BbqmyTjMza49H7GZmmXGwm5llxsFuZpYZB7uZWWYc7GZmmXGwm5llxsFuZpYZB7uZWWYc7GZmmXGwm5llxsFuZpYZB7uZWWYc7GZmmXGwm5llxsFuZpaZKn7zdB9JP5T0gKT7JH28ioaZmVlnqvihjWHgkxFxh6R5wEpJN0TE/RXUbWZmbariN0/XRcQd6f5zwAPAwm7rNTOzzlQ6xy5pMXAYcHuDsqWSBiUNDg0NVblbMzMrqSzYJe0IfBv4bxHxm/ryiFgWEQMRMdDf31/Vbs3MrE4lwS5pBkWoXx4RV1VRp5mZdaaKq2IEfAV4ICL+pvsmmZlZN6oYsb8F+APgbZJWpds7K6jXzMw60PXljhFxG6AK2mJmZhXwJ0/NzDLjYDczy4yD3cwsMw52M7PMONjNzDLjYDczy4yD3cwsMw52M7PMONjNzDLjYDczy4yD3cwsMw52M7PMONjNzDLjYDczy4yD3cwsM1X9NN4SSQ9JeljS2VXUaWZmnanip/F6gQuB44BDgJMlHdJtvWZm1pkqRuxvAB6OiEciYiPwdeDECuo1M7MOVBHsC4HVpcdr0rIxJC2VNChpcGhoqILdmplZI1UEe6PfO40tFkQsi4iBiBjo7++vYLdmZtZIFcG+Btin9Hhv4IkK6jUzsw5UEew/AQ6QtK+kmcBJwPIK6jUzsw70dVtBRAxL+q/AdUAvcHFE3Nd1y8zMrCNdBztARFwDXFNFXWZm1h1/8tTMLDMOdjOzzDjYzcwy42A3M8uMg93MLDMOdjOzzDjYzcwy42A3M8uMg93MLDMOdjOzzDjYzcwy42A3M8uMg93MLDMOdjOzzDjYzcwy01WwS/qCpAcl3S3pO5IWVNUwMzPrTLcj9huAQyPid4CfAud03yQzM+tGV8EeEddHxHB6uILih6zNzGwKVTnH/iHg2grrMzOzDkz4m6eSbgT2aFB0XkR8N61zHjAMXD5OPUuBpQCLFi3qqLFmZjaxCYM9Io4Zr1zSqcDxwNEREePUswxYBjAwMNB0PTMz686EwT4eSUuAs4AjI+LFappkZmbd6HaO/QJgHnCDpFWSLqqgTWZm1oWuRuwRsX9VDTEzs2r4k6dmZplxsJuZZcbBbmaWGQe7mVlmHOxmZplxsJuZZcbBbmaWGQe7mVlmHOxmZplxsJuZZcbBbmaWGQe7mVlmHOxmZplxsJuZZcbBbmaWGQe7mVlmKgl2SZ+SFJJ2raI+MzPrXNfBLmkf4Fjg8e6bY2Zm3apixH4+8BkgKqjLzMy61NVvnko6AVgbEXdJmmjdpcDS9PB5SQ91s+9JsCvw5FQ3ogK59APcl+kol37AttGXV7WykiLGH2hLuhHYo0HRecC5wNsj4llJjwEDETHdn5iWSBqMiIGpbke3cukHuC/TUS79gLz6MuGIPSKOabRc0m8D+wK10frewB2S3hARv6y0lWZm1rKOp2Ii4h5gt9rj3EbsZmbbKl/H3tyyqW5ARXLpB7gv01Eu/YCM+jLhHLuZmW1bPGI3M8uMg93MLDPbdbBLWiLpIUkPSzq7QfksSd9I5bdLWjz5rWxNC33575Lul3S3pJsktXQ97FSYqC+l9d6bvspiWl6i1ko/JP1e+rvcJ+mfJ7uNrWrh9bVI0g8l3ZleY++cinZORNLFktZLurdJuST9Xern3ZJeN9ltrEREbJc3oBf4ObAfMBO4Czikbp2PAhel+ycB35jqdnfRl7cCc9L9j2zLfUnrzQNuBVZQXI015W3v4G9yAHAnsHN6vNtUt7uLviwDPpLuHwI8NtXtbtKXI4DXAfc2KX8ncC0g4HDg9qlucye37XnE/gbg4Yh4JCI2Al8HTqxb50Tgq+n+lcDRmugjtlNjwr5ExA8j4sX0cAXF5w6mo1b+LgD/A/gr4OXJbFwbWunHh4ELI+JpgIhYP8ltbFUrfQlgp3R/PvDEJLavZRFxK/DUOKucCFwahRXAAkl7Tk7rqrM9B/tCYHXp8Zq0rOE6ETEMPAv81qS0rj2t9KXsdIpRyXQ0YV8kHQbsExHfn8yGtamVv8mBwIGSfiRphaQlk9a69rTSl88C75e0BrgG+NjkNK1y7f5fmpa6+q6YbVyjkXf9tZ+trDMdtNxOSe8HBoAjt2qLOjduXyT1UHzx3GmT1aAOtfI36aOYjjmK4gjqXyUdGhHPbOW2tauVvpwMXBIRfy3pTcDXUl82b/3mVWpb+T8/ru15xL4G2Kf0eG+2PHwcXUdSH8Uh5niHcVOllb4g6RiK7/g5ISI2TFLb2jVRX+YBhwK3pE87Hw4sn4YnUFt9fX03IjZFxKPAQxRBP9200pfTgW8CRMSPgdkUX6q1rWnp/9J0tz0H+0+AAyTtK2kmxcnR5XXrLAdOTfffC9wc6QzLNDNhX9L0xd9ThPp0ncuFCfoSEc9GxK4RsTgiFlOcLzghIganprlNtfL6upripDbpR2oOBB6Z1Fa2ppW+PA4cDSDpNRTBPjSprazGcuAD6eqYw4FnI2LdVDeqbVN99nYqbxRnwH9Kccb/vLTsLyiCAooX57eAh4H/B+w31W3uoi83Ar8CVqXb8qluc6d9qVv3FqbhVTEt/k0E/A1wP3APcNJUt7mLvhwC/IjiiplVFN/6OuXtbtCPK4B1wCaK0fnpwBnAGaW/yYWpn/dM19fWRDd/pYCZWWa256kYM7MsOdjNzDLjYDczy4yD3cwsMw52M7PMONjNzDLjYDczy8z/B/1EX6n+2alIAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f45807792b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ipywidgets import interact\n",
    "\n",
    "def plot_fit(degree=1, Npts=50):\n",
    "    X, y = make_data(Npts, error=1)\n",
    "    X_test = np.linspace(-0.1, 1.1, 500)[:, None]\n",
    "    \n",
    "    X_poly = PolynomialFeatures(degree=degree).fit_transform(X)\n",
    "    X_test_poly = PolynomialFeatures(degree=degree).fit_transform(X_test)\n",
    "\n",
    "    model = LinearRegression()\n",
    "    model.fit(X_poly, y)\n",
    "    y_test = model.predict(X_test_poly)\n",
    "\n",
    "    plt.scatter(X.ravel(), y)\n",
    "    plt.plot(X_test.ravel(), y_test)\n",
    "    plt.title(\"mean squared error: {0:.3g}\".format(mean_squared_error(model.predict(X_poly), y)))\n",
    "    plt.ylim(-4, 14)\n",
    "\n",
    "    \n",
    "interact(plot_fit, degree=[1, 40], Npts=[2, 100]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detecting Over-fitting with Validation Curves\n",
    "\n",
    "Clearly, computing the error on the training data is not enough (we saw this previously). As above, we can use **cross-validation** to get a better handle on how the model fit is working.\n",
    "\n",
    "Let's do this here, again using the ``validation_curve`` utility. To make things more clear, we'll use a slightly larger dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAGspJREFUeJzt3X+MZWV9x/HPd2cHnVVk1rK2MrosJgpVCKxMGppNrEhTiD83KKLxd2k3amLVWpK1NsH+SNh0q9YmTduNtWq1iqJZqdiidTG2RNBZF0WFba26wEBlDSytMMCwfPvHnVnu3jnn3uf8fs4571dCmL1zZu5zzrnzPc/5Pt/nOebuAgC037qmGwAAKAcBHQA6goAOAB1BQAeAjiCgA0BHENABoCMI6ADQEQR0AOgIAjoAdMT6Ot/s5JNP9i1bttT5lgDQevv37/+5u2+atF2tAX3Lli1aWFio8y0BoPXM7FDIdqRcAKAjCOgA0BEEdADoCAI6AHQEAR0AOoKADgAdUWvZIgBUae+BRe2+7qDuOrKkU2ZndPmFp2v71rmmm1UbAjqATth7YFHv/cItWlo+KklaPLKk937hFknqTVAn5QKgE3Zfd/BYMF+1tHxUu6872FCL6kdAB9AJdx1ZyvR6F5FyAdAJp8zOaDEheJ8yOyOpH/l1eugAOuHyC0/XzPTUca/NTE/p8gtPP5ZfXzyyJNfj+fW9BxabaWxFCOgAOmH71jldefFZmpudkUmam53RlRefpe1b53qTXyflAiCTmFMX27fOJbalL/l1eugAgrU1dbGaRw99va0I6ACCtTV1MS6/3iWkXIBAMaca6tLW1MXqeer6+SOgAwGYhTgwqTQwZmn59S4h5QIEaGuqoWx9SV20FT10IEBbUw1l60vqoq0I6ECANqcaytaH1EVbkXIBApBqQBvQQwcCkGpAGxDQgUCkGhA7AjqA1mOOwIC5e21vNj8/7wsLC7W9H4D2yRqcR+cIrNq4YVpXvOx5nQjsZrbf3ecnbUcPHUA08kzgSpojIEn3Pbjcu8lfE6tczOyjZnaPmX1/6LWnmtlXzey/Vv6/sdpmAhhn74FFbdu1T6ftvFbbdu2LfrGsNHkmcI2bC9C3yV8hZYsfk3TRyGs7JX3N3Z8t6Wsr/wbQgJAVENsS8PNM4Jo0F6BPk78mBnR3/4ake0defoWkj698/XFJ20tuF4BAk3q1bVryNs8yt0lzBEJ/tmvyTiz6ZXe/W5JW/v+08poEIItJvdrQNEYMvfg8E7hWn1Q0OzO95nt9m/xV+aCome2QtEOSNm/eXPXbYQLKu7pn0rIEIWmMWFaTzDOBa/Uzff/SsmZnpmUmHXlwuZef77wB/Wdm9nR3v9vMni7pnrQN3X2PpD3SoGwx5/uhBLH80aJcl194+pqyveGeacg6NON68cOfjTo6BFkmcI1+po8sLWtmekofuvScNb+jD52ZvCmXayS9aeXrN0n6YjnNQZVYArY9sqQ/xj0cWQpLY2TpxceUi8+SToqt7VWY2EM3s09LeqGkk83sTklXSNol6bNmdpmk2yVdUmUjUQ6WgG2HPHdS43q1IWmMkF78H//zD4J68XUK/UyH3oG03cSA7u6vTfnWBSW3BRVjCdh2qCL4TEpjTErb7D2wqPseXE782SY7BKGf6b50Zlg+t0fyLgEbQ/VDn1QZfNLO5aS0zbi0XJMdgtDPdJ5yyDZi6n+P5K0gYCC1XlXdSU06l+N68eMuJk2WBYZ+psfdgVQxWNrUACwBvWeyLgHbl9xjTCalP/Iqci7TLjKzM9ONfw5CPtNpgV9S6R2WJjtBBHSM1ZfcY0zKfJjGcE8xrWY45FymXWTe//LnZW5TU5IC/7Zd+0rvsDTZCSKgYywGUptRxsM00paVHTW7Ye0My6T2SN17YlMVHZYmO0EEdIxV1e0/qjHcI19npqMBzzv4xUOPau+BxaC0RdsD+KgqOixNdoKocsFYk6ofEI/RyTMhwVySlh/z3k4uq+Lh300+UJweOibqYs+si9Ie9BCir2MiVaSSmkxPEdDRS11b12PvgcXE2/xQoemArh03qZoOS1OdIAI6eqdrtfWr+5NXaDqga8ctr5gvagR09E5dZWV1/eGPS7XMTE/plefO6apv36Hlo2tz6nMZ2tXnOQmr53LxyJJMOlYCGttFjYCO3qmjrKzO3uy4dl958VmSpKu+dcdxr0+vM+2+5OxMS8y2eU5CkYvr6LkcvSzGdFGjygW9U8e6HnUuVZzW7rnZGW3fOqfd1x3U8mPHh6GkypZJS8y2dT2Uokvnhgw2x3JRI6Cjd+ooK6uzNztpf8pYYjbkfWKV5+I6vIhZyGBzLBc1Ui7onTrKytIml5yU8NzLYXlSA5P2p6wlZts6WzQtIKe9HjrDdlVMFzUCOnqp6rKyyy88XZd/7rtrUh0PPJI+K7NI3n3c/oTO9g0J/G2ckzCVMmN2yixx+yz1/CbplefGc0xIuSBaoeuwx7he+/atc3ryE9f2l5aPps/KLDPvPnxMdl93UK88d27ibN+2plTSrB6DtBmzaa9nSYu5pOtvO5yneZWgh47SlFmmF9pbjbk2+kjGJ/yUlXdPOiaf3784ccmGtqZUkoSkTebGDPJmmaQVy4CoREBHScoOrKE1z7HURiddzNICg2uwbGue53qGKHJMqkyp1DkhZ1LaZPTOY7htIatPDotlQFQi5YKSlF2mF9pbjaE2Oq0s7vwzNq1JYaxKKp0rK+URwzEZVbR0MKtx+zqachptW9qzU5PElpIioKMUZQaRvQcWtS5lwGq0NxRDbXTaxez62w4fW6kyyegFr6yVLWM4JqPqrMuXxtfm37DzRRPv8tJs3DAd9cqjpFxQirLSBau9pbQBqwcePr5KpO712pPSBuMuZqspjNN2Xpv4xKDRny0j5RHjGvZ13zVkOQahbZiZntIVL3teVAF8FD10lKKsdMGk3tKRpeXjbtXrXK89LW2QlnMdvpjV2WtOOyaSGqsGqvuuIcvnIq0NsffGk9BDRynKqpAI6S2NDvDVVRudljZ4wvp1mpmeGtsbPP+MTfrkjbev+Z3nn7GpkraOHpOmq4GauGsI/VyktS323ngSAnpHxLCkZxmBNbRkrO5Bz9WV9pLcv7SsD116ztjjn1arnPR6Feey6WqgmEsiY25bVgT0Dmi691WGtOVJ01Q5wDccUE+amdYDjzyauPTscFsmXcxCc8hVncsmKl+SLkw37HxRZe9XRBtnwCYhh94BdVcQlG04Ny0NgvlqjcuTTlhb9lf1oOdwnvzI0vLYYB7altAcclXnsu4cdt1lihggoHdArHXHoQNwSUHMNRiUeiwhlq4GuCqCQ5YStiwDZaGDxlWdy7IGrUPPawydjBiXhKgaKZcOKKtksCxZ0wZpwWrcBI+q0kqhgXO1njlUaJ62qnNZRp44y3ltupPRhTRkHoUCupm9W9LvaNChukXSW9z9oTIahnCx1R1nHYDLunZGyO/MK6QteY9tSJ62ynNZNE+c5bw23cloehC4KblTLmY2J+n3JM27+5mSpiS9pqyGIVydtdghsvbO0tIBsxPWDh/3O/NKasv0OtPGDdO1HNvYzuWwLOe16ZUbm75DaErRlMt6STNmtixpg6S7ijcJeRTpfWUtk5u0fdbeWVo6QNLEFfPK7vHFUMIWa8VFlvPa9HFs+g6hKbkDursvmtlfSLpd0pKkr7j7V0prGWqRNdcYsn2etMG4IJZWzlhVj6+sgBrD3IAyZT2vacexjuPSRBoyhvNtnrJmxsQfNNso6fOSLpV0RNLnJF3t7p8c2W6HpB2StHnz5nMPHTpUqMEo17Zd+xJ7MmmDfmnbb9wwrQ0nrD/2YT7/jE26/rbDpX64Y/iDCZW0HvfM9FQ06ZO8ip6DOo9LnZ+XqvfLzPa7+/zE7QoE9EskXeTul638+42SznP3t6f9zPz8vC8sLOR6P1QjbdEok/STXS8J3n5UF4JXEVkvlH3R1eNS9X6FBvQidei3SzrPzDaYmUm6QNKtBX4fGpB1wkloDrJNE5uq0NdBuUm6elxi2a/cAd3db5J0taTvaFCyuE7SnpLahZokVnVMmR54+NHECRlJ26dp+x9pETGuSR6Drh6XWPar0ExRd7/C3c9w9zPd/Q3u/nBZDUM9RsvkNm6Ylnww5T1pynZSWV1aeWGWD3MbZvVlaWPTZXux6upxiWW/cufQ8yCHHr88ucCkASGT9LrzNuvPtp818T3bMICYp41tGsStU1ePS5X7VfmgaB4E9PhlHSRd9Ud7b9Gnbrx9TVlhSFBuw0BZG9qI7goN6KzlguPknZBx/W2H11wIQqdajxtQiqU3F8ugVwxiOSdYi9UWcZy8ucAiAS/tYnHSzHQ0S7DGMujVtC4ui9uG8ZtQBHQcJ+9aIkUCXtpFxEyNL8G6KpZBr6bFsCxukrxBuWsXKFIuWCPP1PciU63T1v1491U3J27fRJqj6bVJYhFj6qnIUrldW5WRgI5SFA14SReRtOd4NpXmiHXRrDrFuOhV3qC898Bi6lLJbR0bIaCjNGUHvNjWeUec5yTPXcNqrz5NW8dGCOgNqbtSIOn9pLhTCKQ54hPjOclz1zDuUYNNX6CKoA69AXVPpEl6v+kpk1xaHnpoZ2yTeVC9LpQg5vl7GrfI3F9eek50x6COxbmQU92VAknvt3zUjwvmVbcB8elKhUeeyqy03vvc7Ex0wTwLUi4NqLtSIMvvbetgELLrUoVH1vGbGMcCykBAb0Bazm92w7S27dpX+u1vlocwxzYY1IWUQKxiLEGsS4xjAWUgoDcgqXcwPWX6xUOP6r4HlyVlq6XN+35JOfSYeihZ6osJ/NnFWIJYpy6WoZJDL1nIjLWknN+TTlhfWU476f12v+ps7b7k7ONee+W5c9p93cFopkCHjjV0JRdclbTPJLNfu4ceeomy9ChHewen7bw28XeWdfub1htZfa3IbLuqhKYEupQLLlvIeeXOpjsI6CUqEliavv2NMSiGHpMu5oLLSiFNOq9dTDv0GSmXEhUJLE3f/sYYFEOPSddWQiwzhRTjeUV1COglKhJY8q5yWJYYg2LoMWn6Yli2MucpxHheUR1SLiVKqiYxSeefsSno58fd/lZdxRFrXW5ISqBrueAye9WxnldUg4Beou1b57Rw6N7jHsXmkj6/f1Hzpz41d4CpY8Cy7UGxS7ngMsdT2n5ekQ1ruZSsimdP8jzLfmnDQ7NRL54p2pAqBqEY2OoXetXIi4BesirKD5suaUT9upRCQn2ocilZFRUXXaviAFANeuglC71dDq1aWd1uafmopsx01F1z3IIDSEBAr8Ck2+XQqpXR7Y66H+uZ1xnMWfgKaAdSLg0InThS94MwkrDwFdAeBPQGhFatxFDdEsNFBUAYUi4TFEk3pP1saNVKDNUtMVxUAIQpFNDNbFbSRySdqcGkyN9292+W0bAYpOW6Fw7dq+tvOzxx0DMtTx46HTuGadsxXFQAhCnaQ/+wpH9191eZ2QmSNpTQpmikpRuGp/anDWim/ez7r/mBnvSE9UFVKzFMMInhogIgTO6AbmZPkfQCSW+WJHd/RNIj5TQrDmlphdHFEpLWDU/72SNLyzqyNHjMXEjVStMTTGK4qAAIU6SH/ixJhyX9g5mdLWm/pHe6+wPDG5nZDkk7JGnz5s0F3q5+WR6uPBrAQ3+26YdIhGj6ogIgTJEql/WSni/pb9x9q6QHJO0c3cjd97j7vLvPb9oUtoxsLJJmaFrKtqM55aSfTcMAI4AyFAnod0q6091vWvn31RoE+M5IesDC687bHDQNP+lnN26YTnwfBhgBlCF3ysXd/8fM7jCz0939oKQLJP2wvKbFISndMH/qU4NyyqM/m7YsatIAI7MzAWRVtMrlHZI+tVLh8mNJbynepPjlzSlnWeel6gdatAEXNSCbQgHd3W+WNHHR9b5KC0iTgtKkJ7X3ARc1IDum/lekyBoozM5kyQEgDwJ6RYoEJJ7UzkUNyIOAXpEiAYkHWnBRA/IgoFekSEBKKnns2wOCuagB2bHaYkWKroHS99mZLDkAZEdAD5ClfG5429kN03rC+nW6f2mZgJRD3y9qQFYE9AmylM+Nbnvfg8uamZ7Shy49h8AEoHLk0CfIUq1CqR2AJhHQJ8hSrUKpHYAmEdAnyFKtQqkdgCYR0CfIUj5HqR2AJjEoOiKpouXKi88KXl1RotQOQDPMffSBatWZn5/3hYWF2t4vq7Tlbfs2qQdAXMxsv7tPXAiRlMsQqlQAtBkBfQhVKgDajIA+hCoVAG1GQB9ClQqANqPKZUieKpXRqpjzz9ik6287TJULgNpR5VJAUlXMKKpkABRFlUsNkqpiRlElA6AuBPQCQqtfqJIBUAcCegGh1S9UyQCoQ28C+t4Di9q2a59O23mttu3ap70HFgv/zqSqmFFUyQCoSy+qXLI8pCKL1Z9911U3p27DgCiAuvSih17llP7tW+c0l5JSmZudIZgDqE0vAnraoORiSYOVTEgCEINeBPS0QUmTUnPpWXLu27fO6cqLz9Lc7IxMg545qRYAdevFxKK9Bxb17qtuVtKezs3O6IadL1qzPcvoAogFE4uGbN86lxjMpeR0DMvoAmijwgHdzKbM7ICZfamMBlUlbeAyKR3DMroA2qiMHvo7Jd1awu+pVJaBS5bRBdBGhQK6mT1D0kskfaSc5lQny8AlVSsA2qjQoKiZXS3pSkknSvoDd39pwjY7JO2QpM2bN5976NCh3O9XJ5bFBRCL0EHR3DNFzeylku5x9/1m9sK07dx9j6Q90qDKJe/71W371rljAbuqmaYAUKYiKZdtkl5uZj+V9BlJLzKzT5bSqshQ9QKgDXIHdHd/r7s/w923SHqNpH3u/vrSWhYRql4AtEEv6tCLouoFQBuUEtDd/etJA6JdQdULgDboxfK5ReV5eDQA1I2AHmi46gUAYkQOHQA6orc99NGJQ6RQALRdLwM6E4UAdFEvUy5MFALQRb0M6EwUAtBFrU+55MmFnzI7k/g8USYKAWizVvfQV3Phi0eW5Ho8Fz7u+Z8SE4UAdFOrA3reXDgPdQbQRa1OuRTJhTNRCEDXtLqHzqJZAPC4Vgd0cuEA8LhWp1xYNAsAHtfqgC6RCweAVa1OuQAAHkdAB4COIKADQEcQ0AGgI1o/KDqMNc4B9FlnAjprnAPou06kXPYeWNR7Pvtd1jgH0GutD+irPfOj7onfZ41zAH3R+oCetOLiMNZ1AdAXrQ/o43rgrOsCoE9aH9DTeuBTZqxxDqBXWh/Q01Zc/MCrzyaYA+iV1pctsuIiAAy0PqBLrLgIAFKBgG5mz5T0CUm/IukxSXvc/cNlNSwEM0MB4HFFeuiPSnqPu3/HzE6UtN/MvuruPyypbWMxMxQAjpd7UNTd73b376x8/X+SbpVUWyRNqj9nZiiAPiulysXMtkjaKummMn5fiLT6c2aGAuirwgHdzJ4s6fOS3uXu/5vw/R1mtmBmC4cPHy76dsek1Z8zMxRAXxUK6GY2rUEw/5S7fyFpG3ff4+7z7j6/adOmIm93nLT6c2aGAuirIlUuJunvJd3q7h8sr0lhqD8HgOMVqXLZJukNkm4xs5tXXvtDd/9y8WaFof4cAB6XO6C7+39IshLbAgAooPVruQAABjox9T8Js0gB9E0nAzqzSAH0USdTLswiBdBHnQzozCIF0EedDOjMIgXQR50M6MwiBdBHrRoUDa1cYRYpgD5qTUDPWrnCLFIAfdOalAuVKwAwXmsCOpUrADBeawI6lSsAMF5rAjqVKwAwXmsGRalcAYDxWhPQpbVBfXVAlKAOAC0L6Cy6BQDpWpNDlyhdBIBxWhXQKV0EgHStCuiULgJAulYFdEoXASBdqwZFKV0EgHStCugSi24BQJpWpVwAAOkI6ADQEQR0AOgIAjoAdAQBHQA6wty9vjczOyzpUAm/6mRJPy/h97QF+9tt7G+3lbG/p7r7pkkb1RrQy2JmC+4+33Q76sL+dhv722117i8pFwDoCAI6AHREWwP6nqYbUDP2t9vY326rbX9bmUMHAKzV1h46AGBEtAHdzC4ys4Nm9iMz25nw/SeY2VUr37/JzLbU38ryBOzv75vZD83se2b2NTM7tYl2lmXS/g5t9yozczNrdVVEyP6a2atXzvEPzOyf6m5jmQI+z5vN7HozO7DymX5xE+0si5l91MzuMbPvp3zfzOyvVo7H98zs+ZU0xN2j+0/SlKT/lvQsSSdI+q6k545s83ZJf7vy9WskXdV0uyve3/MlbVj5+m1d39+V7U6U9A1JN0qab7rdFZ/fZ0s6IGnjyr+f1nS7K97fPZLetvL1cyX9tOl2F9znF0h6vqTvp3z/xZL+RZJJOk/STVW0I9Ye+q9J+pG7/9jdH5H0GUmvGNnmFZI+vvL11ZIuMDOrsY1lmri/7n69uz+48s8bJT2j5jaWKeT8StKfSvpzSQ/V2bgKhOzv70r6a3e/T5Lc/Z6a21imkP11SU9Z+fokSXfV2L7Sufs3JN07ZpNXSPqED9woadbMnl52O2IN6HOS7hj6950rryVu4+6PSrpf0i/V0rryhezvsMs0uNq31cT9NbOtkp7p7l+qs2EVCTm/z5H0HDO7wcxuNLOLamtd+UL29/2SXm9md0r6sqR31NO0xmT9G88l1gdcJPW0R8txQrZpi+B9MbPXS5qX9BuVtqhaY/fXzNZJ+pCkN9fVoIqFnN/1GqRdXqjB3de/m9mZ7n6k4rZVIWR/XyvpY+7+ATP7dUn/uLK/j1XfvEbUEq9i7aHfKemZQ/9+htbekh3bxszWa3DbNu6WJ2Yh+ysz+01J75P0cnd/uKa2VWHS/p4o6UxJXzezn2qQc7ymxQOjoZ/nL7r7srv/RNJBDQJ8G4Xs72WSPitJ7v5NSU/UYM2Trgr6Gy8q1oD+bUnPNrPTzOwEDQY9rxnZ5hpJb1r5+lWS9vnK6EMLTdzflRTE32kQzNucX5Um7K+73+/uJ7v7FnffosGYwcvdfaGZ5hYW8nneq8HAt8zsZA1SMD+utZXlCdnf2yVdIElm9qsaBPTDtbayXtdIeuNKtct5ku5397tLf5emR4fHjBq/WNJ/ajBa/r6V1/5Egz9safAB+JykH0n6lqRnNd3mivf33yT9TNLNK/9d03Sbq9zfkW2/rhZXuQSeX5P0QUk/lHSLpNc03eaK9/e5km7QoALmZkm/1XSbC+7vpyXdLWlZg974ZZLeKumtQ+f3r1eOxy1VfZ6ZKQoAHRFrygUAkBEBHQA6goAOAB1BQAeAjiCgA0BHENABoCMI6ADQEQR0AOiI/wfh7kJG+oBv3QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f4580760390>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X, y = make_data(120, error=1.0)\n",
    "plt.scatter(X, y);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import validation_curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def rms_error(model, X, y):\n",
    "    y_pred = model.predict(X)\n",
    "    return np.sqrt(np.mean((y - y_pred) ** 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import make_pipeline\n",
    "\n",
    "def PolynomialRegression(degree=2, **kwargs):\n",
    "    return make_pipeline(PolynomialFeatures(degree),\n",
    "                         LinearRegression(**kwargs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "degree = np.arange(0, 18)\n",
    "val_train, val_test = validation_curve(PolynomialRegression(), X, y,\n",
    "                                       'polynomialfeatures__degree', degree, cv=7,\n",
    "                                       scoring=rms_error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's plot the validation curves:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEKCAYAAADuEgmxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJztvXmcHVWZ//9+7tZrlu50ErJvECB7QhPQYCCCmQTHBQQNow7gKMqos311dJYfOOj8Rh230ZFh0GF0HEQEWfwqO6K4ECAJCQQCZE86ayed7vTed3m+f5y63XVv3+7cdPou3f28X6/qqjrnVNXTde+tT53znPMcUVUMwzAMIxsChTbAMAzDGDqYaBiGYRhZY6JhGIZhZI2JhmEYhpE1JhqGYRhG1phoGIZhGFljomEYhmFkjYmGYRiGkTUmGoZhGEbWhAptwGBSU1OjM2fOLLQZhmEYQ4aNGzceU9Xx2ZYfVqIxc+ZMNmzYUGgzDMMwhgwisvd0ylvzlGEYhpE1JhqGYRhG1phoGIZhGFkzrHwahmGcGdFolLq6Ojo6OgptijHIlJaWMnXqVMLh8Bmdx0TDMIxu6urqGDVqFDNnzkRECm2OMUioKsePH6euro5Zs2ad0bmsecowjG46OjoYN26cCcYwQ0QYN27coNQgTTQMw0jBBGN4Mlifq4mGYRiGkTUmGgCJRKEtMAwDaGxs5Pbbbx/QsVdeeSWNjY39lrnlllt46qmnBnR+w2GiARDrANVCW2EYI57+RCMej/d77COPPMLYsWP7LXPbbbdxxRVXDNi+XHGq/62YMNEAQCHaXmgjDGPE8/nPf56dO3eyZMkSPvvZz/LrX/+aVatW8Sd/8icsXLgQgPe+971ccMEFzJ8/nzvvvLP72JkzZ3Ls2DH27NnD+eefz8c+9jHmz5/P6tWraW93v+8bbriB+++/v7v8rbfeyrJly1i4cCGvv/46APX19bzjHe9g2bJlfPzjH2fGjBkcO3Ysxc54PM4NN9zAggULWLhwId/85jcB2LFjB1dccQWLFy9m2bJl7Ny5E1Xls5/9bHfZe++9FyDj//a///u/LF++nCVLlvDxj3+ceDze57UKhXW5TRLrgEh5oa0wjKLhn/7vq7x28OSgnnPe5NHc+q75feZ/+ctfZuvWrWzevBlwD9YXXniBrVu3dncVveuuu6iurqa9vZ0LL7yQ973vfYwbNy7lPNu3b+eee+7he9/7Hu9///v52c9+xoc+9KFe16upqWHTpk3cfvvtfO1rX+P73/8+//RP/8Tb3/52/u7v/o7HHnssRZiSbN68mQMHDrB161aA7maxD37wg3z+85/nqquuoqOjg0QiwQMPPMDmzZvZsmULx44d48ILL2TlypUAKf/btm3buPfee/n9739POBzmz//8z7n77ruZP39+xmsVCqtpJLGahmEUJcuXL08ZW/Dtb3+bxYsXc/HFF7N//362b9/e65hZs2axZMkSAC644AL27NmT8dxXX311rzK/+93vWLduHQBr1qyhqqqq13GzZ89m165dfPrTn+axxx5j9OjRNDc3c+DAAa666irADaYrLy/nd7/7Hddddx3BYJCJEydy6aWX8uKLL/b6355++mk2btzIhRdeyJIlS3j66afZtWtXxmsVEqtpJInZCFjD8NNfjSCfVFRUdG//+te/5qmnnuK5556jvLycyy67LOPYg5KSku7tYDDY3TzVV7lgMEgsFgPcQLhTUVVVxZYtW3j88cf57ne/y09/+lO+9a1vZSzb3/n8/5uqcv311/Mv//IvvcqlX+uuu+46pY25wmoaSaymYRgFZ9SoUTQ3N/eZ39TURFVVFeXl5bz++uusX79+0G245JJL+OlPfwrAE088wYkTJ3qVOXbsGIlEgve973188YtfZNOmTYwePZqpU6fy0EMPAdDZ2UlbWxsrV67k3nvvJR6PU19fz7PPPsvy5ct7nfPyyy/n/vvv5+jRowA0NDSwd+/ejNcqJFbTSJKIQjwGQbslhlEoxo0bx4oVK1iwYAFr167lne98Z0r+mjVruOOOO1i0aBHnnnsuF1988aDbcOutt3Lddddx7733cumllzJp0iRGjRqVUubAgQPceOONJLzu+snawY9+9CM+/vGPc8sttxAOh7nvvvu46qqreO6551i8eDEiwle/+lXOOuusbsd7knnz5vGlL32J1atXk0gkCIfDfPe736WsrCzjtQqFZFMVGyrU1tbqgCZh6mqFY2/CuLOhZNSpyxvGMGXbtm2cf/75hTajoHR2dhIMBgmFQjz33HPcfPPN3Y75oU6mz1dENqpqbbbnsNdqP9EOEw3DGOHs27eP97///SQSCSKRCN/73vcKbVJRYaLhJ2Z+DcMY6Zxzzjm89NJLhTajaDFHuB9zhhuGYfSLiYYf63ZrGIbRLzkTDRGZJiLPiMg2EXlVRP4yQ5kPisjL3vIHEVnsy9sjIq+IyGYRGYB3ewBowvk1DMMwjIzk0qcRA/6Pqm4SkVHARhF5UlVf85XZDVyqqidEZC1wJ3CRL3+VqqYGfck1sXYIl+b1koZhGEOFnNU0VPWQqm7ytpuBbcCUtDJ/UNXkyJn1wNRc2ZM1VtMwjCFFZWUlAAcPHuSaa67JWOayyy7jVN3xv/Wtb9HW1ta9n02o9ZFIXnwaIjITWAo830+xPwMe9e0r8ISIbBSRm3JnXRrWg8owhiSTJ0/ujmA7ENJFI5tQ64Wg0GHUcy4aIlIJ/Az4K1XNGDJTRFbhRONzvuQVqroMWAt8UkRW9nHsTSKyQUQ21NfXn7nBVtMwjILxuc99LmU+jS984Qt8/etfp6Wlhcsvv7w7jPnDDz/c69g9e/awYMECANrb21m3bh2LFi3iAx/4QErsqZtvvpna2lrmz5/PrbfeCrggiAcPHmTVqlWsWrUK6Am1DvCNb3yDBQsWsGDBgu4YU/2FYPdz3333sWDBAhYvXtwd3TYej/OZz3yGhQsXsmjRIr7zne8ALmjh0qVLWbhwIR/5yEfo7OzstuW2227jkksu4b777mPnzp2sWbOGCy64gLe97W3do8szXWvQUdWcLUAYeBz4m37KLAJ2AnP7KfMF4DOnut4FF1ygA6KzRfXApp4lHh/YeQxjiPPaa6/17DzyOdW7rhzc5ZHP9Xv9TZs26cqVK7v3zz//fN27d69Go1FtampSVdX6+nqdM2eOJhIJVVWtqKhQVdXdu3fr/PnzVVX161//ut54442qqrplyxYNBoP64osvqqrq8ePHVVU1FovppZdeqlu2bFFV1RkzZmh9fX33tZP7GzZs0AULFmhLS4s2NzfrvHnzdNOmTbp7924NBoP60ksvqarqtddeqz/60Y96/U8LFizQuro6VVU9ceKEqqrefvvtevXVV2s0Gu22qb29XadOnapvvPGGqqp++MMf1m9+85vdtnzlK1/pPufb3/52ffPNN1VVdf369bpq1ao+r+Un5fP1ADboaTzXc9l7SoD/Arap6jf6KDMdeAD4sKq+6Uuv8JzniEgFsBrYmitbe2FNVIZREJYuXcrRo0c5ePAgW7ZsoaqqiunTp6Oq/P3f/z2LFi3iiiuu4MCBAxw5cqTP8zz77LPd82csWrSIRYsWdef99Kc/ZdmyZSxdupRXX32V1157ra/TAC5U+lVXXUVFRQWVlZVcffXV/Pa3vwWyC8G+YsUKbrjhBr73ve91Ny099dRTfOITnyAUcn2RqqureeONN5g1axZz584F4Prrr+fZZ5/tPs8HPvABAFpaWvjDH/7Atdde2z1Z06FDh/q81mCTy95TK4APA6+ISDJwy98D0wFU9Q7gFmAccLvTGGLqYqBMBB700kLAj1X1sRzamkq0AyIVpy5nGMOZtV8uyGWvueYa7r//fg4fPtw9r8Xdd99NfX09GzduJBwOM3PmzIwh0f14z48Udu/ezde+9jVefPFFqqqquOGGG055Hu0nPl82IdjvuOMOnn/+eX75y1+yZMkSNm/ejKr2sq+/60BPGPVEIsHYsWMzxsPKdK30CarOlFz2nvqdqoqqLlLVJd7yiKre4QkGqvpRVa3y5dd66btUdbG3zFfVf86VnRmxmoZhFIx169bxk5/8hPvvv7+7N1RTUxMTJkwgHA7zzDPPsHfv3n7PsXLlSu6++24Atm7dyssvvwzAyZMnqaioYMyYMRw5coRHH+3pe9NXWPaVK1fy0EMP0dbWRmtrKw8++CBve9vbsv5/du7cyUUXXcRtt91GTU0N+/fvZ/Xq1dxxxx3dc3g0NDRw3nnnsWfPHnbs2AG4iLmXXnppr/ONHj2aWbNmcd999wFObLZs2dLntQYbiz2VCXOGG0bBmD9/Ps3NzUyZMoVJkyYBbhrVd73rXdTW1rJkyRLOO++8fs9x8803c+ONN7Jo0SKWLFnSPX/F4sWLWbp0KfPnz2f27NmsWLGi+5ibbrqJtWvXMmnSJJ555pnu9GXLlnHDDTd0n+OjH/0oS5cu7XM2wHQ++9nPsn37dlSVyy+/nMWLF7NgwQLefPNNFi1aRDgc5mMf+xif+tSn+O///m+uvfZaYrEYF154IZ/4xCcynvPuu+/m5ptv5ktf+hLRaJR169axePHijNcabCw0OvSERk8SCMFZCwfPMMMYIlho9OHNYIRGt9hTQDSeSE1IxCAeLYwxhmEYRYyJBhCLK7FEWo3LIt4ahmH0wkQDUJTOWFr3NBMNY4QynJqsjR4G63M10fBo7UoTDetBZYxASktLOX78uAnHMENVOX78OKWlZx6MdcT3nurqivKZb/4Xi6dV8fHVS3oyrAeVMQKZOnUqdXV1DEpIHqOoKC0tZerUM48JO+JFIxIK8PX4V/jN4VWATzRiHaAKGQYIGcZwJRwOM2vWrEKbYRQx1jwVCHKg9GzO6thFaicqtZn8DMMw0jDRAFqr5zFX93CoKU0kzBluGIaRgokGEJmyhHLpZMfuXakZVtMwDMNIwUQDGH/OhQC0HNyWmmE1DcMwjBRMNIDqqefSSiklJ7anZlhNwzAMIwUTDSAQClEXnsXE9p2kdE+Pd0GisFMrGoZhFBMmGh4nR5/NObqXo82dqRnWRGUYhtGNiYZHZMK5lEkXO3ftSM2wJirDMIxuTDQ8zpo5D4DmA6+nZlhNwzAMoxsTDY/xk2fSQhnhE2+mZlhNwzAMoxsTDY9AMMB+zxmegtU0DMMwujHR8NE06hzmJPZyvMVXu9A4xLoKZ5RhGEYRYaLhIzxhLqUSZVe6MzzaVhiDDMMwioyciYaITBORZ0Rkm4i8KiJ/maGMiMi3RWSHiLwsIst8edeLyHZvuT5XdvpJOsOb6tJGhptfwzAMA8htaPQY8H9UdZOIjAI2isiTqvqar8xa4BxvuQj4D+AiEakGbgVqAfWO/bmqnsihvZw1ZQbNlBFuSK9pmF/DMAwDcljTUNVDqrrJ224GtgFT0oq9B/gfdawHxorIJOCPgCdVtcETiieBNbmyNUkwGGBfaDYT2m2shmEYRiby4tMQkZnAUuD5tKwpwH7ffp2X1ld6pnPfJCIbRGTDYMw21jTqbGYn9nGy1TcyPNYJiUTfBxmGYYwQci4aIlIJ/Az4K1U9mZ6d4RDtJ713ouqdqlqrqrXjx48/M2OB4IRzKZEou3b7x2vYhEyGYRiQY9EQkTBOMO5W1QcyFKkDpvn2pwIH+0nPOWfNPB+Axv1pI8NNNAzDMHLae0qA/wK2qeo3+ij2c+BPvV5UFwNNqnoIeBxYLSJVIlIFrPbScs6UKTM4qeWEGtLCpJsz3DAMI6e9p1YAHwZeEZHNXtrfA9MBVPUO4BHgSmAH0Abc6OU1iMgXgRe9425T1YYc2tpNKBRgb3g249vMGW4YhpFOzkRDVX9HZt+Ev4wCn+wj7y7grhyYdkoaK8/mohO/oLWjk4rSEpdoNQ3DMAwbEZ6J4IS5RCTGnl0+Z3giCvFY4YwyDMMoAkw0MjDRGxl+Yn/6yHCrbRiGMbIx0cjAjKnTadIKgsfTwqRHza9hGMbIxkQjA6FQgN3h2dS0pYdJt8CFhmGMbEw0+qCx8hxmxPfR1ekfGW41DcMwRjYmGn0QGD+XiMTZs+uNnkQTDcMwRjgmGkCmnsETvJHhx/0jwzVhfg3DMEY0JhoAoVLShWP21Gk0aiWBY+lzhlsPKsMwRi4mGgCBAITLU5Ii4QC7QhlGhltNwzCMEYyJRpJIRa+kE5VnMy2+n3iX3xluNQ3DMEYuJhpJMohGYPy5hCXO3t0+Z7jVNAzDGMGYaCSJVPZKGj/jPAAa9vmc4XGbkMkwjJGLiUaSYAiCJSlJZ0+bRoOOQo69kVrWmqgMwxihmGj4SWuiKo04Z/i4VnOGG4ZhgIlGKhmaqBoqzmZqfD/qFwqraRiGMUIx0fCTwRku488lJAnqzBluGIZhopFCuBQCqfNSjZ/uRoYf2+cLk241DcMwRigmGumk1TbmzpjCMR0N/pHhiRjEo3k2zDAMo/CYaKQTThWN8kiAncHZVPdyhluYdMMwRh4mGulk8Gs0VJ7N1Nh+1D9PuPk1DMMYgeRMNETkLhE5KiJb+8j/rIhs9patIhIXkWovb4+IvOLlbciVjRmJVJAevFBq5hIU5ci+7T2J5tcwDGMEksuaxg+ANX1lquq/quoSVV0C/B3wG1Vt8BVZ5eXX5tDG3oj0qm3UeM7w+r2v9iRaTcMwjBFIzkRDVZ8FGk5Z0HEdcE+ubDlt0kTj3OmTqdcxaH3ahEyqeTbMMAyjsBTcpyEi5bgayc98yQo8ISIbReSmvBuVJhqjSoPsCM6mqsU/Z7jaTH6GYYw4Ci4awLuA36c1Ta1Q1WXAWuCTIrKyr4NF5CYR2SAiG+rr6wfHonBvZ/jx8rOZHK+DFGe4+TUMwxhZ9CsaIhIUkX/NsQ3rSGuaUtWD3voo8CCwvK+DVfVOVa1V1drx48cPjkXBkDebn+86NXMJohzbb3OGG4YxculXNFQ1DlwgIr0n0R4ERGQMcCnwsC+tQkRGJbeB1UDGHlg5Ja2Jany3M/y1nkSraRiGMcIInboILwEPi8h9QGsyUVUf6O8gEbkHuAyoEZE64FYg7B17h1fsKuAJVW31HToReNDTqRDwY1V9LKv/ZjCJVELb8e7d82ZO5shvxpI46hsZbjUNwzBGGNmIRjVwHHi7L02BfkVDVa871YlV9Qe4rrn+tF3A4izsyi1pNY2xpQF+F5jDrBbfWI14FyTiEAjm2TjDMIzCcErRUNUb82FI0REqgUAYEj0xpo5XzOGtLZtcCJFwuUuMtkNJ75DqhmEYw5FT9p4Skaki8qA3uvuIiPxMRKbmw7iCEylP2dVx5xJAaaqzJirDMEYm2XS5/W/g58BkYArwf7204U/apEzV088F4Ig5ww3DGKFkIxrjVfW/VTXmLT8ABqlva5GT5teYN30yh7Sa+FH/hEwmGoZhjByyEY1jIvIhb8xGUEQ+hHOMD3/C5SA9t6imIsB2mcXYZn/gQmueMgxj5JCNaHwEeD9wGDgEXOOlDX9Eeo0OP1ZxNhPjh6DL6yWscYh1FsA4wzCM/NNv7ykRCQLvU9V358me4iNSAV3N3buJcecQaFVaD71JxYylLjHa7npbGYZhDHOyGRH+njzZUpyk9aCq9kaGH9njc4ZbE5VhGCOEbAb3/V5E/h24l9QR4ZtyZlUxkdaDav70szio1cSOmDPcMIyRRzai8VZvfZsvTUkdIT58CQQhVNY9U9/EyiDPyGzmtfjmDLeahmEYI4RT+TQCwH+o6k/zZE9xEqlImd61vvxsJrZtgK4WVxOJdUIiAYFiiDRvGIaRO07l00gAn8qTLcVL2niNxLi5AHQcTo4MtwmZDMMYGWTzavykiHxGRKaJSHVyybllxUSaX6Nq6nmAOcMNwxh5ZOPTSI7J+KQvTYHZg29OkRKKpAQvnD9jInV/qCFqznDDMEYY2US5nZUPQ4qeSAV0NAIwZVSAp5nNIhsZbhjGCCObKLflIvKPInKnt3+OiPxx7k0rMnxNVCJCffkcJsQOQ6c38M9qGoZhjACyjXLbRU/X2zrgSzmzqFhJc4bHPWd41xHPGZ6IQjyWb6sMwzDySjaiMUdVvwpEAVS1HcjJnOFFTbgsJXhh0hmeMmd4zGobhmEMb7IRjS4RKcM5vxGROcDIi9CXFrxw3rTx7E+Mp8uc4YZhjCCyEY1bgceAaSJyN/A08Lc5tapY8TVRzRwbZJvMYtRJnzPcRMMwjGFONr2nnhSRTcDFuGapv1TVYzm3rBjxiYaIcLTsbGo6XnDO8JJR0HkSVF2txDAMYxiSVdwLVT2uqr9U1V9kKxgicpc3r/jWPvIvE5EmEdnsLbf48taIyBsiskNEPp/dv5IHIhX43Tnx6nMAeoIXJmLQ1lAAwwzDMPJDLoMl/QBYc4oyv1XVJd5yG3TP4fFdYC0wD7hORObl0M7sCQSdQ9xjjOcMP75vW0+Z1vp8W2UYhpE3ciYaqvosMJDX7uXADlXdpapdwE8opjk9fE1U86eNZ29iAh2HX+/Jj7X3jN0wDMMYZmQzuG+OiJR425eJyF+IyNhBuv5bRGSLiDwqIvO9tCnAfl+ZOi+tOPCJxpzqIK+R5gwHq20YhjFsyaam8TMgLiJnA/8FzAJ+PAjX3gTMUNXFwHeAh7z0TF5k7eskInKTiGwQkQ319Xl4WPu63QZEOFJ2NtWxo9DR1FOmo8nmDTcMY1iSjWgkVDUGXAV8S1X/Gph0phdW1ZOq2uJtPwKERaQGV7OY5is6FTjYz3nuVNVaVa0dP378mZp1akIRCEa6d2PVbmR4/OgbqeWstmEYxjAkG9GIish1wPXAL7y08JleWETOEnF9U0VkuWfLceBF4BwRmSUiEWAd8PMzvd6g4muiGjP1XAAa9r+eWqatARLxfFplGIaRc7IJjX4j8Angn1V1t4jMAv73VAeJyD3AZUCNiNThBgmGAVT1DuAa4GYRiQHtwDpVVSAmIp8CHgeCwF2q+upp/2e5JFIJ7ScAmD9lHHuen0jocJpoaNwJR2Ueaj+GYRh5IpvBfa8Bf+Hb3w18OYvjrjtF/r8D/95H3iPAI6e6RsHw1TTm1oR4nFm8tWl773Kt9SYahmEMK7LpPfXHIvKSiDSIyEkRaRaRk/kwrmgJl4EEAQgFhMOlc6iK1XfPt9FNvBPaGzOcwDAMY2iSjU/jWzh/xjhVHa2qo1R1dI7tKn58tY2o5wxPpDvDAVpHZsQVwzCGJ9mIxn5gq+dvMJL4neFTnDO8qe713uW6mi2QoWEYw4ZsHOF/CzwiIr/BFxJdVb+RM6uGAj7RmDelml0vnkXpodepylS25ShUzcibaYZhGLkim5rGPwNtQCkwyreMbMI9wQvPqwnzqs6iIpMzHFxPK5vVzzCMdIZgA042NY1qVV2dc0uGGoEAhMsh2kpJSDhYOocx0eecQJSl1zcU2o7BqLMKYqphGAUmHnXN1LGOnnWsA4IlUD3bDRoeImQjGk+JyGpVfSLn1gw1Ik40ALqq5sJR0Po3kOkX9y7begwqJ9pcG4YxnEkkXNDSaEfqOtFHS0OsHY696YQjUp5fWwdINqLxSeBvRaQTN0+4AGo9qHB+DS9cyJgp59J1JEjb9vWMzSQaiairhZRX59lIwzAGHVUXXy5dIOIDiDmXiMLxHVA1E0qL/7Har0/DC/MxX1UDqlpmXW7TiFR2b86fUsWjiYso3/Nk372lLB6VYQxdEgn34tewGw6/DPXb4MQeaDnsgpQORDCSaBwadkHr8UEzN1f0KxpeN9sH82TL0CMYdm2SwPwJYX6i7yASbyX6xpOZy0fboLMljwYahnFGqDpBOLEHjmx1645G0EQuLgZN++DkoRyce/DIpvfUehG5MOeWDFW8rrelIWHKnIVsS8wgvvWBvntFtB7No3GGYZw2qtBxEhr3weFXXA2g/YSrDeSDlsNOnIq0Z1U2orEKeE5EdorIyyLyioi8nGvDhgy+JqoPLargh/F3UHpyD7FDfdyijpMQ68qTcYZhZE1nCzTudzWKhp3Qdjx/QpFO+wk4vrMoI2Vn4whfm3MrhjK+Hg/n1ZRwaMJKmht+TOjlBwhNXpzhAHW+jTHFMxmhYYxYulpdfLiORogX2ctcV7PXs2pOUXXJzSbK7d58GDJkSQYv1Dil4QBXzR/LT399KTfsewJtPY5UjOt9TNtxGDXJjfUwDCO/RNvdm3x745k5r/NBrKPouuTaU2sw8IUUefusMp4pW02QOJ1bH85cXuNOOAzDyA+qrmfS0deh/nVoOVL8gpEkEYXj21OnlC4gJhqDgc+vMao0zFvmz+Q38UXotl/0PajHut8aRu5JxKH5CBx51fVMig3R4KGacF19i6BLronGYOCraYjA1eeV87PAasq6Gujc8WzmY+KdRfPmYBjDjngUmg44sWg+6N7WhzzJLrkHC2qFicZgEC4nGbwQYHxlhLFzLqROa+h8uZ9hLi1W2zCMQSXaASf2OrFoPVq43k+5pOVIQbvkmmgMBsnghR6hgLBuwSh+HL+C0Q2vED22M/NxNteGYQwOnS2ui2r9NmhvAIpzjMOgUcAuuSYag4WviQpgTk0phya/g04N0/bSA30fZ74Nwxg47Y1wbLtzFHeOsFmok11y8zzuy0RjsEgTjZJQgHctnMgvEhdTuvdp4h2tmY+zuTYM4/To7gm1DU7shq4RHJon2SU3j8+QnImGiNwlIkdFZGsf+R/0Rpi/LCJ/EJHFvrw93sjzzSKyIVc2Diq+HlRJlk8r5fej1lCS6KDt1UcyH6cJN9eGYRj906snVEehLSoOEtG+e2nmgFzWNH4ArOknfzdwqaouAr4I3JmWv0pVl6hqbY7sG1yCIQiVpiRVRkIsXTCfzYnZxF99GE30FY/qWNHGmTGMvKHaM1lRZ7Orhbceg+bDXniP4dQTauiSTRiRAaGqz4rIzH7y/+DbXQ9MzZUteSNS0evtZ+3Z5dz+4mpu6biDtr0bKJ+VIfajzbVhDAcSCS/6q7q1Jtdx9yacSK5jmfdzEjnWGGxyJhqnyZ8Bj/r2FXhCRBT4T1VNr4V0IyI3ATcBTJ8+PadGnpJIZa+R3tXlESrmXkrD6/9LdPMDmUVjN0ZyAAAfVUlEQVQD3BuViYaRbxJx93afiLmXl3hyHXUPe1WvFpwuBGniMNx7KxndFFw0RGQVTjQu8SWvUNWDIjIBeFJEXlfVjKPkPEG5E6C2traw39zSMSCBlDemQACuXjiWB7at4sb6X9LeeJiysRnmCo+2um6DJb19I4Zx2vgf/omYt86wb2/3xmlSUNEQkUXA94G1qtr9iq6qB731URF5EFgO9DG0uogIBKF0rNdPvIcpY0o4PHUNcuAXnHzpIcpWfSLz8a31JhpGKom0ph2NZ5dmb/5GjiiYaIjIdOAB4MOq+qYvvQIIqGqzt70auK1AZp4+5eN6iUYkGGDNkpk8s38JF+16lOiKGwlHSnof29Hk+lwXURhkI0fEo26O6XiXb4m6h77fB2APf6PIyJloiMg9wGVAjYjUAbcCYQBVvQO4BRgH3O6mIifm9ZSaCDzopYWAH6vqY7myc9ApqXS9qNIc4vMmlPKdsWu5vPn/5/i2XzFucaZpStR1vx09OT+2GrkjHnPxxeJd7kXALw6xTkwMjKFKLntPXXeK/I8CH82QvgvINHvR0KG8Bk7WpSZFgpy/eDm7n51IySsPkVi4NvN0Gq3HoPIsm2tjKJBIOF9UtL13rcF8BcYwxZ5MuaC82jnE07h0ZgWPlaxmctsbNB/YlvlYjfdq3jKKBFU301vzYTi2A468Asd3wMkDrobYedLVME0wjGGMiUYuSDrE0xhdFqbk3NW0a4TjG/uJR9VU58I6F+H8wCOOrjZoOeqCwx1+xYVsaD7k4v6YOBgjEBONXFFR0ytJBNYumMCjsoIpR39D88kTfRysLqzz0deKYtKVEUW0wzURNuz2ROINV5PoPDk8w2wbxmliopErIhUQKuuVXFMZ4ci0d1JCF8de+mX/50jEXIydo6+7sArG4BPrgrYGNz/B4a0utHbTfuhozGs8H8MYKhR8cN+wpqLGPYB8hAPCpcvmsXHvXKbt/AUdb1lHaeQUH0Os3bWdl46B0VMglKG77nAmkfANTIu6fbRndDL+UcvZrPHiHHUNnXmiDaNIMNHIJWVVrmkjre17dnUpd1ev4YLGb7P/zeeYtuBt2Z2vowk6TkLlhOHRw6rfUcvx1HAWhmEUBSYauSQQdMKRFo+qNBxgzrLLqH/6h7RveZjovEsIB6SPk6SjbrrHtgY3nmMoxKvqanPNa9FWJwJJkbCxCoYx5DDRyDXl43qJBsAFU0fzaMnlXNP6IMeP7GP8pBmnd95EFBr3utAjo6cUV/iRWJcTic6TboIc8w0YxrBhiLdvDAEiFSnzhyepLAkRmnclCRUOvfCQa6YfCNE2N9Vlw+68T/vYTSLupt1s3A9HXoOj3iQ55kw2jGGHiUY+KB/XK0kEVs6fxm8Dtcw48hQnW/uYDjZbOhpdz5+Thxi4AmWJqovIe/IQ1L/puqae2O0GuJlj2TCGNSYa+aCsKuMI8aqyEo5OfydjaGbvpifO/DqagJbDbnxH2yCPKo+2Q0u9N8jtZVe7aTns/BTmmzCMEYP5NPJBHw7xYAAuql3Ozj1TGLXzF7S85V1Unqr7bTYk/R3NhyEQIuWh3mta2bT9TPmasGYmwzAAq2nkj/LeI8QBJo0p4eXqNcyO7aRu+yuDe814pxdQr61nibWnLR2pS7wzbekywTAMoxsTjXwRKc/oEC8JBZhRu4YWLaVly8N0xiyekWEYxYs1T+WT8hrXqyiNuVOqWF+2krc1P8ORo0eZPjnDdLCGYQwNYp3QfsK3NEBb2n5yG4EZb4FZl8LUWggW/wRsJhr5pHuEeOoI58pIiND8d1Oy8Qn2vPhLJr/rI4SyHuxnGEbeiHW6kD4Nu1xnk3QRaDvhdQ7JQKQCyqrdc6BqFkxe5pqM9/we3nzctUTMeCvMWgnTlrvJ3IoQE418Egh4DvFjvbIWnzeXzS/N45wjj3K8eR0Tx1QUwEDDMLpRdS95R1+Do9vc+vjOVB9fyWj3my6rgnHnwLTqnv2yKicS5VVuqoS+YsbFo3BwE+x+Fnb/FnY85QRj2kUw+1KYdrFr3i4STDTyTfm4jKIxpizC0envZMmef+XXm35P1+K3Ma4yQnkkWAAjDWME0nHSjXU68ppbH33dRTUA9xAffx4svBYmzIOac9xvORg+8+sGw04gpl0El/w1HHoZdv/GE5HfuPypy52AzHgrRAob/cFEI98kHeLRtpTkQAAWXHgpR/d8nzG7f0nH/LdyoLGd8kiI6oowZWETD8MYNOJRV2tI1iLqt7nJzwAQqJoJMy9xAjHhfLcfyMNvMBCCKcvc8ta/gCOv9gjI3t97+bWuCWvmChf5Os+YaBSCPhzi40dX8Pvq1VzWcC+Pv/IHZs2/mLauGG1dMSpLQlRXRCgJWYc3wzgtEjFo3Oem6D32phOIY2864QDXhDThfJi7FibOg5pzi6M5KBCESYvc8pZPOnFL1j6eXQ+/DcDkpc6JfsE4qJqeF7NEew3mGsSTi9wF/DFwVFUXZMgX4N+AK4E24AZV3eTlXQ/8o1f0S6r6w1Ndr7a2Vjds2DBY5ueORAKObM0Y8nv73gOMfvwvmMhxdsl0Ds18DxPmr4JQBAEqS0OMqyghHDRHuWH0ItrmahDHd8Cx7W59YnePQAQjUDPX1SAmerWIigkurs9QQdVFZNj1GycgTXXOt/K3uwbUXCYiG1W1NuvyORaNlUAL8D99iMaVwKdxonER8G+qepGIVAMbgFrckOWNwAWq2tf8qMAQEg1wwf0y+Dbau+Lsq29k/5ZfMbPu55zNfhoYw/4pV1K58J0kSsciwJiyMFUVEetlZYxc2o672sPxHe4henwHNB2gO8pByWioOds5qMed7Zax07woCcMEVSeKiTgsev+ATnG6opHTu6eqz4rIzH6KvAcnKAqsF5GxIjIJuAx4UlUbAETkSWANcE8u7c0rFTUZRaMsEqS8rIwZtWuJLv0jHty6kYm7H+atB+6h88D97JuwitCC99LIdE52RBlbHmFsWYSgtVoZw43k7IrRNjcBWcMuTyC8WkS77x1y1CQnCues7hGIivFDqwYxEESgejaMPz9vlyy05E4B/POh1nlpfaUPH8JlEK7I2Kd7XGUJR5tdtNgFiy8kuqCW/3l9N5XbH+bKI89QevQJDo5ZRmLeVTRMWEJTW5SqighjSsNDfjI/Y4ijCfegj3W4UP2xDvfQ72pz3/WuVrfd5YW36bVuSc1PD2EjQaie6cYxdNcg5kDJqIL8uyORQotGptcA7Se99wlEbgJuApg+PT+OoEGjogYae4tGeSTIzHHlNHfEONHWBSR4y/zZRM/7K36w/UPwxiO8r/EJxj/3/9FYPoOOue/l+LTLaGyPUF1ewujS0LB/wTLOgOQbfPqDOv2h3h2TrKsnLlnMW/zb3fsdPb6DbAiG3YtTpLxnXXmW610YSUuPVLoeTFUzhsSo6eFMoUWjDpjm258KHPTSL0tL/3WmE6jqncCd4HwauTAyZ5SOBanrcw7sUaUhRpWGaOmKcaIlCsS5/LzxdJ3zp/x4xzU0bnuG97f8kvM3/xvtr/6Q1jnv5PisK2msqKK6PMKo0kJ/vMagoeoFoGz3ljbftm8/1t7z0M/49u6lZxOEUoJufEIo4gamhUrdAztU6h7i5eMgWOLllaRuJ48LlvRMRBap8IlBuT38hyg5dYQDeD6NX/ThCH8n8Cl6HOHfVtXlniN8I7DMK7oJ5wjvd5KIIeUIT9JU56ZszYK2rjgNrV20R53IdMWVR3d2sfO1jVwbf4S3BzcTlwgt01fROOc9UD2T8kiQskiQslDI/B6DiSZcm3pLPXQ1e/Oex93DOOPi5fVXLtbpHvrRDEus3V0zGyToe0BXZHhoV3hv8P63+Yq0h3rF4AxcM/LD+PMhPLCwI0XlCBeRe3A1hhoRqQNuBcIAqnoH8AhOMHbgutze6OU1iMgXgRe9U912KsEYspSPy1o0yiNByiNltEfjnGiNQleM98wtoWvOW3hkZy3ffW0X74s+wvv2/ooZex+ndcIFNE9bxZGzlpMIl1MaClBqInJqVN0c561HocVbktut9d76mJu3ZCBIwPXgSV9CJZ6vqwzKqyHkbYfLe9JTtn37oTKIeOtgZPg7gI2CkfOaRj4ZkjUNcFOm9hXkrB86YwkaWrto6Yx5+8ovd3bx+LZjXBl9ihsiT1OjDSQkRNvEZbRMXkHrWReRiFQiuLDsw0JEVF0TXyLes07Zjvn2fdudLZ4Y1PcWhlhH6jUkCJXjXZ/+ygmuZ07lBLdfOhoCYTcYyy8CQb8o+PIyzOJoGGdEHmsaJhrFQFuDm2lvgHTFE5xo7aK5I4bixOMXO7t44PUOpnZs58rg87w7/AIT9LgTkAlLaJ28gpZJF5OIuF4nBRGRWKdrb+9sdkv3dkv/6dG23mJwRoh7s08KQlIMKn3CUFaVnzAShjEQTDQGxpAVjX5GiJ8O0bjS2NZFU3sUBVSVnY0J1h+Isf5AF6WN21kbfIF3h55nMvUkJEhbzSJap1ziBKSkJ46NE5EgpZEApaEgEujp0ibelnT/cWn+FhEhAc2HCTTuRU7sRRp3IycP9ohAVzNyip42GipFSka57pSRSrcuqXRNMoGQe/vvfoMP+vaDfeSlbYfLnDCU15yy/T7TzySZZt2cjYJjojEwhqxowGk5xE9FNKE0tUVpau8i4ft4G9oTPH8wxvoDUTqPbOdyeYF3Bp9nhhwhQYDmcQvpnLqClklvIV5ald3FNEGorZ5I8z5KmvcSObmPSPM+Is37CcQ7e2wqHUe0cjLxyGgS4Qri4UoSkUoS4Uri4QpvnZqWPnI3H630A/01BEUIBoWACEERAgEIBsQtEiAQgJAIAS/NRvIbg4qJxsAY0qIR7XCB1AaRRAJau2I0d7igh/5PujOmbD7qBOTEgZ1cHHuedwaeZ3bgEAkCNI6dR3TaJbROeSvx0mpQJdTuxCFych8lzfuInNxLpKWOQKy9+7yx0mq6Rk2nc9R0ukZPp2vUDLpGTSNR4HDOxUgwKSLihEQks//anyQivdL8tb1MnI5PvM+ikr576pOml5A+DOnrTL2Ki7tnIhAQJ9AiEECstmeiMTCGtGiAC43Q1ZKTU8cSSktHjObOGB3R1Gaw7masuiiHDuxmfstzrA08z9zAARIITaVTqYweIxzvEYfO8FhaK6fTXjmdjsrpdHoCISWVhAIQCrgftmHkA4EeMQkIAVxCUmSkW3B6vpMiPcemnKh7s3+RE5Hu6/oz/eeV5FkyNOOKuP1kzfSMMNEYGENeNM7QIZ4t0bjS0hmjpSNKR6x33/9kM9b+fXuY2vAHFrKDfTqB7TqV7YkpvKlTaeTUYRsCAkGBsNdUE/L8IgHx/8iT2+K2BQK+PHzlXb73Q02/2Cne0DPuS+q5k+dPbvvt6Cnns9MrUxISxpYIY0uFMSVuGVsijCkNUBnu+w3bMJII7jcSCLgaVMireQa9mmjAq40GAhCUgNsW6emsMlzGaRinSelYCBzIbrTuGRAOClXlYarKw3TGEk5AOmN0eQJSXRZg7ZwIzJlLZ+wcdjbGqYzDQoXzE0o8AdEExBMQUyWWwFuUuJKyH0tAXJ2fJZ5wzuME3tp7X0koKOrWvvyU7WQ51RQ/TV+kF8noyO4+p1u7ayXctu+ayTJx7/pKctvltceU9j4+slAARkfSBSWQQWCEspD0Ek0hs5j2iGiP6BpDF8X9XrL6cvtIvshMqY5TkqexmCYaxUQg4CaEaT2at0uWhAKUhCKMq4jQEU3Q3BmlpTNGLK5evjCvxr4mp6IrrjR2KE2dSmOn0tiRoKmzZ7+pU2nqUN5sjdPYGaNtgOMC+yK95tadnhQVr5D4yuPL69GcniaX1BqYqzUKdDf/+GthqTU2SSkv/uumpYn0vib4/g/fectCQnlYqAjjrd2STPPvl4VGhpC6FxjN+FKUK+xpUGyUj8uraPgpDQcoDZcwvrKEtq6414QVIz6MmjBzRSQoTKgQJlRkV74rrpz0CUpjh9IZ1141MUV926k1o5TaF6m1N6WndpXc9n+KyX1NFsC7lu84f80vpUaWobbVY49mqJ31vl7CM0Lp+R/TbfXbHFfoiCqtUaUtm7BZQLknJOU+MSkPQTDQU4NLX7od636h9DVF+pswQ4GeptdwwJUNec2wPYtvX6RXeiTobIwMoUnVTDSKjXApREa5eEYFxIUsCTK+soSOaJx48ofd/aDo+cHje2Dgz0t5+GjKAyq5kfIgS659DzF/Rl/apfSRcQo09fR5JRIUasqFmiKYVXSokVClPQqt3SKitHY5MXFrt9+att/UmeBwixOghPaIsV8Uk02QvYWyRxhzQThA71pTpKc2VRGGypT9nvSKiDAhFmdgHo3Tx0SjGBk9GVqOuIFwZzza+cwQcRNDDXf8b8OJbiH0vQH79lPeyL2HSVwTxONKXJVYXEl4a6ulDT4BESoi7mGZb5I+tYRCTH2+vG4/n7q1pvr1eue59M44tHni1xp14tbiCWJjc8KJXxa1q7GPP8fmW/8oL/fARKMYiZRD9SwvcN5JN2tZx8mBB8gzTol/jEQwteX/jFB1nQUSCSWWSJDofpAkiHsPm0QiQSzhhCZ5TK/znLElxmAgIgQFgniRV11qzq8bT7jOFn5xcdsurWritFOfZJAw0ShmRKB0jFvAxV7qaHKLb7S1UbyIQNhrFC9hcEag+Ws/kN7Ep77t9APTd1MT+guV4m+GTO6nNCdq6jE9eb2bIXub1cuwjLuJhKu5Jby39gRK3KvVDXdRDQaEyohrosrEtHPzN7GpicZQoqTSLWOmuDkWkgISbSu0ZUYe8Q8Oy5B7OmcaBGuKg0TC9SJKqLoanCcuyf2edZqQ+XxsPYLorTXVr+YX0EL6wwqNicZQJTmfwqiz3HScSQHpamFkfpWNkUwg4Ho9FUIIE4lkzzBN9X1l8oN5Y2lTOpN0+0k01UGv2pOXKJ4alYnGcCAU8cJ4j4d4zPODNHqO9CxnezMMY0D0hAAZPF9Yf8QTrrOGerUpVSWSx8lwTDSGG8GQmxuivNoJSNN+JyCGYQwLgoFkZw3PIw+nF5XyDBnpsSGHN8GQ64VVNbNXmHHDMIyBYE+SkUBZlRsweLIO2k8U2hrDMIYwVtMYKQRDrsZRNcvNZ20YhjEArKYx0igb66ZOtVqHYRgDIKc1DRFZIyJviMgOEfl8hvxvishmb3lTRBp9eXFf3s9zaeeIw2odhjFMEAiXQyB/oX5yVtMQkSDwXeAdQB3wooj8XFVfS5ZR1b/2lf80sNR3inZVXZIr+wxcraNklJufvL2h0NYYhnEqAmEXZihcAZEKTzDy62XIZfPUcmCHqu4CEJGfAO8BXuuj/HXArTm0x8hEIAhVM5yANO63+FaGUTSIG8CbFIdIBYRKCm1UTkVjCrDft18HXJSpoIjMAGYBv/Ill4rIBiAGfFlVH8qVoQYuvtWESqt1GEahSNYiIpVOJApQi8iGXIpGptEmfY2CXwfcr5oSB3y6qh4UkdnAr0TkFVXd2esiIjcBNwFMnz79TG0e2XTXOqrcoMB4V6EtOg18YWqz3QZvxLy6tSZ6R98z8oyABLyww4GeBXHTICdihZsuQIJuvFMwnMGHkOFxl3HAXVpaIOTVJipdZIchQC5Fow7wx+udChzso+w64JP+BFU96K13icivcf6OXqKhqncCdwLU1tbar30wKB0NkfPg5AFoOz7Ak4j7YSV/aIFg6kMg/aGQfDD0Su+nHORmJGx34KAMgpKS5stLxCAe7XmwJfcLPB/KoNL9GWT4LHt9vsEMn2EGIej1WWfxeapmuN9x17Tq30/m9/cZBELuDT8Qch1Eurc9YUjuB0JF+dZfCHIpGi8C54jILOAAThj+JL2QiJwLVAHP+dKqgDZV7RSRGmAF8NUc2mqkEwjC2OlQOhaaD3lpIffj7haBYM+PKRBKFYg89uYYdLofXoPwkMj4gItlTlPtebD6BTRFJL2Hq/+NvNd2prXvf/Pn9Zfmf/gX0wNTxD3Ug1n2/PN/BhrvEYqgjTgYCDm7a6oaE5FPAY/jIqTcpaqvishtwAZVTXajvQ74iSbjFTvOB/5TRBK4X+6X/b2ujDxSOtotxsA43QecMfjYZzCoSOqzemhTW1urGzZsKLQZhmEYQwYR2aiqtdmWL6I6p2EYhlHsmGgYhmEYWWOiYRiGYWSNiYZhGIaRNSYahmEYRtaYaBiGYRhZY6JhGIZhZI2JhmEYhpE1w2pwn4jUA3sHeHgNcGwQzckHQ83moWYvmM35YqjZPNTshb5tnqGq47M9ybASjTNBRDaczqjIYmCo2TzU7AWzOV8MNZuHmr0weDZb85RhGIaRNSYahmEYRtaYaPRwZ6ENGABDzeahZi+YzfliqNk81OyFQbLZfBqGYRhG1lhNwzAMw8iaEScaIrJGRN4QkR0i8vkM+SUicq+X/7yIzMy/ld22TBORZ0Rkm4i8KiJ/maHMZSLSJCKbveWWQtiaZtMeEXnFs6fXBCfi+LZ3j18WkWWFsNNnz7m++7dZRE6KyF+llSn4fRaRu0TkqIhs9aVVi8iTIrLdW1f1cez1XpntInJ9gW3+VxF53fvsHxSRsX0c2+/3KI/2fkFEDvg++yv7OLbfZ0uebb7XZ+8eEdncx7Gnf49VdcQsuBkEdwKzgQiwBZiXVubPgTu87XXAvQW0dxKwzNseBbyZwd7LgF8U+t6m2bQHqOkn/0rgUdzcohcDzxfa5rTvyGFc3/Wius/ASmAZsNWX9lXg897254GvZDiuGtjlrau87aoC2rwaCHnbX8lkczbfozza+wXgM1l8b/p9tuTT5rT8rwO3DNY9Hmk1jeXADlXdpapdwE+A96SVeQ/wQ2/7fuBykWxmux98VPWQqm7ytpuBbcCUQtgyyLwH+B91rAfGisikQhvlcTmwU1UHOkg0Z6jqs0BDWrL/+/pD4L0ZDv0j4ElVbVDVE8CTwJqcGeojk82q+oSqxrzd9cDUfNiSDX3c42zI5tmSE/qz2Xt2vR+4Z7CuN9JEYwqw37dfR++HcHcZ74vdBIzLi3X94DWTLQWez5D9FhHZIiKPisj8vBqWGQWeEJGNInJThvxsPodCsY6+f2DFdp8BJqrqIXAvGcCEDGWK+X5/BFfrzMSpvkf55FNec9pdfTQBFus9fhtwRFW395F/2vd4pIlGphpDevexbMrkFRGpBH4G/JWqnkzL3oRrSlkMfAd4KN/2ZWCFqi4D1gKfFJGVaflFd48BRCQCvBu4L0N2Md7nbCnW+/0PQAy4u48ip/oe5Yv/AOYAS4BDuOaedIryHgPX0X8t47Tv8UgTjTpgmm9/KnCwrzIiEgLGMLDq6qAgImGcYNytqg+k56vqSVVt8bYfAcIiUpNnM9NtOuitjwIP4qrufrL5HArBWmCTqh5JzyjG++xxJNm0562PZihTdPfbc8b/MfBB9RrX08nie5QXVPWIqsZVNQF8rw87ivEeh4CrgXv7KjOQezzSRONF4BwRmeW9Va4Dfp5W5udAsnfJNcCv+vpS5xqvPfK/gG2q+o0+ypyV9LmIyHLcZ3o8f1b2sqdCREYlt3FOz61pxX4O/KnXi+pioCnZxFJg+nwrK7b77MP/fb0eeDhDmceB1SJS5TWtrPbSCoKIrAE+B7xbVdv6KJPN9ygvpPnbrurDjmyeLfnmCuB1Va3LlDnge5wP734xLbieO2/iejr8g5d2G+4LDFCKa57YAbwAzC6grZfgqrgvA5u95UrgE8AnvDKfAl7F9dZYD7y1wPd3tmfLFs+u5D322yzAd73P4BWgtgi+F+U4ERjjSyuq+4wTtENAFPdm+2c4f9vTwHZvXe2VrQW+7zv2I953egdwY4Ft3oFr/09+p5O9FScDj/T3PSqQvT/yvqcv44RgUrq93n6vZ0uhbPbSf5D8/vrKnvE9thHhhmEYRtaMtOYpwzAM4www0TAMwzCyxkTDMAzDyBoTDcMwDCNrTDQMwzCMrDHRMIzTxIt6+plC22EYhcBEwzAKgIgEC22DYQwEEw3DyAIR+QdvroSngHO9tDki8pgX7O23InKeL329iLwoIreJSIuXfpm4+VF+jBsshoh8SERe8OYz+M+kmIjIahF5TkQ2ich9Xvwxwyg4JhqGcQpE5AJcWIiluFg+F3pZdwKfVtULgM8At3vp/wb8m6peSO/4Q8txI2/nicj5wAdwQeOWAHHgg15Mq38ErlAXTG4D8Dc5+wcN4zQIFdoAwxgCvA14UL04SSLyc1y4mbcC9/mmWynx1m+hZ16LHwNf853rBVXd7W1fDlwAvOidowwXcPBiYB7wey89Ajw36P+VYQwAEw3DyI70eDsBoNGrIZwOrb5tAX6oqn/nLyAi78JNmnTd6ZtpGLnFmqcM49Q8C1wlImVeVNB3AW3AbhG5FrrnPV/slV8PvM/bXtfPeZ8GrhGRCd45qkVkhnf8ChE520svF5G5g/5fGcYAMNEwjFOgbsrde3ERWX8G/NbL+iDwZyKSjBKanN7zr4C/EZEXcPO8N/Vx3tdwvosnRORl3DSsk1S1HrgBuMdLXw+cl4N/zTBOG4tyaxiDjIiUA+2qqiKyDrhOVfMyX7Rh5BrzaRjG4HMB8O/epE2NuLksDGNYYDUNwzAMI2vMp2EYhmFkjYmGYRiGkTUmGoZhGEbWmGgYhmEYWWOiYRiGYWSNiYZhGIaRNf8PWTbcoweEpUMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f457f6379b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_with_err(x, data, **kwargs):\n",
    "    mu, std = data.mean(1), data.std(1)\n",
    "    lines = plt.plot(x, mu, '-', **kwargs)\n",
    "    plt.fill_between(x, mu - std, mu + std, edgecolor='none',\n",
    "                     facecolor=lines[0].get_color(), alpha=0.2)\n",
    "\n",
    "plot_with_err(degree, val_train, label='training scores')\n",
    "plot_with_err(degree, val_test, label='validation scores')\n",
    "plt.xlabel('degree'); plt.ylabel('rms error')\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the trend here, which is common for this type of plot.\n",
    "\n",
    "1. For a small model complexity, the training error and validation error are very similar. This indicates that the model is **under-fitting** the data: it doesn't have enough complexity to represent the data. Another way of putting it is that this is a **high-bias** model.\n",
    "\n",
    "2. As the model complexity grows, the training and validation scores diverge. This indicates that the model is **over-fitting** the data: it has so much flexibility, that it fits the noise rather than the underlying trend. Another way of putting it is that this is a **high-variance** model.\n",
    "\n",
    "3. Note that the training score (nearly) always improves with model complexity. This is because a more complicated model can fit the noise better, so the model improves. The validation data generally has a sweet spot, which here is around 5 terms.\n",
    "\n",
    "Here's our best-fit model according to the cross-validation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD9CAYAAACoXlzKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl8VNX9//HXhxAwIBAEZAkEUBFxK9iUahEXUHFF3Fq3al1KF9v6bb+iuHyr31a/Yqlbf1pbqlZpbQUVEVxABLdaQUEiiohV9gAhIgGFCAk5vz9uJiRhljuZO2vez8djHklmbu49NzP5zJnP+ZxzzTmHiIjkjlbpboCIiARLgV1EJMcosIuI5BgFdhGRHKPALiKSYxTYRURyjO/AbmaPmtkmM/uwwX0TzexjM1tiZs+aWWFymikiIn7F02N/DDi1yX1zgMOdc0cCnwA3BtQuERFpJt+B3Tn3BvBFk/teds7V1P04H+gdYNtERKQZgsyxXwm8FOD+RESkGVoHsRMzuxmoAZ6Iss1YYCxA+/btv3nIIYcEcWgRkRZj0aJFnzvnusXaLuHAbmaXA2cCI12UhWecc5OASQAlJSVu4cKFiR5aRKRFMbPVfrZLKLCb2anADcDxzrkdiexLRESCEU+54z+Bt4GBZrbOzK4CHgA6AHPMrNTM/pSkdoqIiE++e+zOuYvC3P1IgG0REZEAaOapiEiOUWAXEckxgZQ7iogEYfriMibOXs76yip6FRYwbtRAxgwpSnezso4Cu4hkhOmLy7hx2gdUVe8GoKyyihunfQCg4B4npWJEJCNMnL28PqiHVFXvZuLs5WlqUfZSYBeRjLC+siqu+yUypWJEJCP0KiygLEwQ71VYoNx7nNRjFxHAy3EPmzCP/uNfYNiEeUxfXJbS448bNZCC/LxG9xXk53HiId24cdoHlFVW4diTe091+7KJAruI1A9cpjN4jhlSxJ3nHkFRYQEGFBUWcOe5R/DqxxXKvcdJqRgRn3I5HRBt4DKV5zhmSNFex/vllNKw2yr3Hpl67CI+ZEKPNpkyeeCyV2FBXPeLAruIL7leipfJwTNS7n3cqIFpalHmU2AX8SGTe7RByOTgGSn3nitpsGRQjl3Eh2ileLkgFCQzdQwhXO5dIlNgF/Fh3KiBjaa7Q+b0aIOi4Jk7FNhFfMj0Hm0miqeKKLRtWWUVeWbsdo4i/Y2bzXdgN7NH8a5tusk5d3jdffsBU4B+wCrgu865LcE3UyT91KP1L54FvZpuu7vu0slaBKz54hk8fQw4tcl944G5zrkBwNy6n0UkSaLNDk33zNGG4qkiCrdtrN+R6OK5NN4bZtavyd1nAyfUff848Brexa1FJGDResFA1B5yqidXxVNFFKuyKFcqj1Ip0Rx7d+fcBgDn3AYz2z+ANkkGyeXZltkmVi842mOpXuc8niqiSNtG+x2JLmV17GY21swWmtnCioqKVB1WEpDrsy2zTbRecLTH/KRFgk7jxFMXf+Ih3SLuJ/Q7mZRmygaJ9tjLzaxnXW+9J7Ap0obOuUnAJICSkhKX4HElBTJl/ZCWwM8no1i94EiPxUqL3DL9A56Yv4bQP2UQPfp4qohe/Th8Ry/PjDvPPQJI/SeObJdoj30GcHnd95cDzyW4P8kguT7bMlP4/WQUrRcc7bFoywVMX1zWKKiHBDFoOWZIEW+NH8HKCWfw1vgREYNwpNdTrXOMGVKU88s5JEM85Y7/xBso7Wpm64BbgQnAVDO7ClgDXJCMRkp6NGe2pXLy8fP7ychPL/i2GUuprKoGYJ98r98WbXLVxNnL9wrqIal6A4/1OovUjrLKKoZNmJfQay1XX6/xVMVcFOGhkQG1RTJMvLMtdTHi5onnk1G4WvqGk3uswf1bdlRz47QPuPPcI7jz3CPCBrBIS+JC6gYtY73OIgV+Y0/6qTmvtVx+vWoRMIko3sWX9JG5eRJZWbFhGgcIm1L535lLI6ZFIh3DIGXLJcR6nYVLMxnhzzWe11ouv161pIBEFc9sS+Xkm6c569A07KXHsmVHNdMXl4V9HsMd24BLji5O+wU2Gj4GjVNQkc47ntdaLr9eFdglMLm+AmKyxLsOTdMUgh+RKpmyZQ2cpoF/2IR5Cb/Wcvn1qsAugcnGFRAzZfDM7yej6YvL+O+p79evp+JXtF5o0+AeSkUE+ndwDqqrYfduaNMG8vJi/04UQbzWsvH16pcCuwQmW3p/IckaPEvWm0WovfEGdYhdyeTr77BzJ2zcCOvXe7cNG7yvmzdDZeXetx07vGBeXQ21tY0PmpcH++wDbdtCu3bQpQt06wZdu3q3vn3hgAP23Dp2bPTrsV5rDZ+Dwnb5OAdbq6obbZdtr9d4mGvGiyRRJSUlbuHChSk/rkhDkT7OFxUW8Nb4Ec3aZ7g0SUF+XiBX/InU3lgaHj/cm04oV996dw29tlVQXLmR4q0bObTqcy7dfzesWAGrV3sBvKnWrb1AXFjo3Tp12vN9u3aQn7/n1rq1F9Crq+Hrr73bzp2wfbu374oK+Pxz2LQJtm1rdJjPO3Thg279WNfnYA45dTjfuuRM6NMn7PnGSlUF9Xykg5ktcs6VxNpOPXZpsZIxeJbM2brNaZdBo6B+4zNL6LClgmM2r+WgzWvZ/sx67qpYQ9/KjfTaVkGe29Oz3pnXGg460OsxDx0KRUXQqxf07Ol97dXL62m3SkJx3ZYtsHIl78xbyGsvzufAitUM2rSSY9+cSv7r/4Qbfwr9+8Pw4XDiiXDaaUxfX+NrQLklzJ5WYJcWKxmDZ9Em04QTT9omUntDF6ZodF/tbvpUbmTA5nWMeflD+MPHDJi7gAXlq+m4a0f9dtvatGNFl94sLBrEmsNOZG1hD9YU9mBNpx607l3Ev246ye+pB6tzZ+jcmbEzP6dy6J6/R5uaag7+fDXHlX/M9e02wYsvwuTJODP6Fh3C6AOH8tLB32HVftGDdi5UvkSjwC5ZJ1Yw9BsskzF4Fm0yTdOSw3hz/OHau2/tLg7btpHu61dy4Oa1HLh5HQdtXkv/LWW03V3jbTQN6NmTyjbdWHj4iXy2X28+7dKHT7v0YdO++4EZBfl5e6ePThvU7L9DEKYvLqufRRuyq3U+H/Y4iA97HMT1E87wBmVLS3nk+vsZ+uFb3PD649zw+uMs6nUIzxwxkucPGc62ffbda9+5UPkSjXLsklKJDizGymHHm+MOYqCz4T46FeTvFYxCmubu48rxf/klLFvGoln/Ztnc+fQsW8GALevo/cVGWtVN1dltrVhT2J1Pu/Thsy59WNOtmFHnn8DxZx8HhYVRjxfKtWfKIKKfks5VE86o/77/+BdwQI9tn3P2stc474N5HLx5DTvz8pl+6Ak88q2z+aRbP8B7k733e4OzMhXjN8euwC4pE8TAYqxgmIwB0WjirSlveB3PUDBqqFPVlxy8eQ1PHd8ZPvrIuy1bBmvX7tkoPx8GDmSudWVJx15e77trH1Z17sXO1m32Ok60tiYykJjMUtFYA8Wd2+Wz+NenRJ6o5RyHl3/Ghe/P5rwP51FQs5M3+w7mL0PP4Y3+R7HqrjMDaWeqafBUMk6iA4vTF5dF/GcP3Z/s2YRNg9mOXTVxTRQqq6zilqdLabd2FeeUf0DHtSs54IsyBmxew0Gb19Jte6W34RNAQQEMGgTHH+99PfRQ73bAAdC6NVeHeWMAr0ca7k0syPK+ZK+zEu35ys8zbj3rsOhvqmZ82OMgbulxEL8/7vtcXDqLy957nslP3cr7fQ+H0zt4f9ccpcAuKZNI0A39E0cSymEnczZhuGAWkXN0/2ozB3yxnn5b1tP/izL6bymj/xfrKa7cSJvaGk6p23Rbm3Z82rUPrx5Qwqr9+zL87OM45qzjoLg4asVJpHNtZUb/8S+EDdxBXZA72Wv1Rxsonnj+NxgzpIhhE+b5elOtLOjIH4/5Ln8Zeg6XLJ3LDQufghNOgJNPhgkT4KijEm5vplFgb+FSOfMykaAb7YLH4C0INXH28qQMiEb6uN+mppqibZvoU7mRPlvLvVvlRvpvWU//LespqN5Zv+3OvHxWdu7Ff7oWM2fA0azcrxe/u/F8XtrZgZveKmdLlTfQWViQz8HHHgb9Yj8H4c4VqK+QSeZqhcn+ZBTpeWyYNvJzrPw8o32b1mytqmb/Lh0ZfMd4Cg75HTz0ENx5J5SUwI9/DLffDvvtF0jbM4ECewuW6mVLk73Y1frKqsDSDdMXl3H3Sx+xe906iivLOWZrOX0qy+m9tS6IV5bT/asv6gcuwav7Luu4P2u7FNHmpJE8u709pft0Y+V+vdjQoSvO9vS+iwoLYPhwdi4u4+uajfX3V1ZV+34Omp5rqzBlj0HXbIeej0gjc0FVm/h5HqP16mudi/7c/+pXcNVVcOut8MADMHUq/P73cPnlYLb39llGg6ctWKoHGqFxoA7VX4cb6AttG+/AZFzt3rXLG5RcvRrWrPG+rl7NpqX/oeo/n9FzawVtamvqN6/F2NChK+sKu7O2U92tsDtb9i9iS/feLKltR8/O7aMOjobcV1eVEeRzEOl4BqxsUEESjp9Pbpk2ozOwweAlS+Caa+Bf/4IzzoBJk7zJVxlIg6cSUzqWLQ39w/n5pBAr/dLUjl01jWvFt22rD9YNA3f9zxs2eHXQDfXsyabWhazsfhAvDBzG2k496gP4+o7dqM7Lb7R5tEASqUfZuV1+zHRCc56D5qa6/H5yi/Z8RHpzbi4/bzSBDQYfeSS8/rrXcx8/Hg47DP74R7go0rWFMl8ggd3MfglcjZfq/AC4wjn3dRD7luRJ17KlfgfeYgW3gl1fU7x1I323bKC4cgO9t26i498r2Mo2OlVs8BaiaqhNG299kb59YdQo72txsfe1b1/o3RvatuWsKD3thmIFs0ipp1vPOqz+5yCfg+aOLyT6fESqwmmueFKEQQ0G06oV/OIXcOqp8IMfwMUXw6uvwv33e9VJWSbhwG5mRcAvgEOdc1VmNhW4EHgs0X1LcjU3EPj92B5pG7+91F6FBWwr/5yDPl9Lvy1eNUlx5Qb6Vm6kb+WGPaWBdba1aUdZp/35oEsPjr1kxN6Bu3t3X+uaRLuQA/j/uO+nRxnkYG9ze7DxPB+p6Agku+ImqoMPhjfegP/5H69iZsECeOop7/4sElQqpjVQYGbVQDtgfUD7lSRqTiDw05uKtU24ANF1+xaO+boc/rjam5CzbBmvvP8hBZ+X129Ti7GhY1fWFPZg7oFDvTVNCnuwurAnqzv3rJ867ienHE2kqwo5/KUc4qk0Cnrp2Ob0YP0G7HB/l/xWxo5dNRHLK5sjFSnCqM9R69Zexczw4XDZZd4CaFOnwimnRN9pBglk8NTMrgXuAKqAl51zl4TZZiwwFqC4uPibq1evTvi4knp+BvsibdO5XT7t8vOw1as4vHwFh5V/xmHln3F4+Wfsv33Lng07dKifkPNhx15MrixgUZtu7O7bl2MPL+KZRWVRc+9BDP42tww0mcv2Jks8bW66fML2XTVU73Yxfy8eyR7Uj+s5Wr0azjoLli6F++6Dn/0srVUzKVtSwMw6A88A3wMqgaeAp51zf4/0O6qKyV7RKj1CvdlfTimt36Zg19cM3vAJ3yz7iJJ1yxi8YTmFX38FQI214j9di1nVewC9RwzjiFHf8WZW9uoV9Z+nYWVN04sapzuIpqPSKAjNeSNL1rkm+80x7nZ/+SVceinMmAE//Sn84Q8JXwGquVJZFXMSsNI5V1F34GnAd4CIgV2yV7T8c1llFf875V1OL1vK4OULGbp2KYduWkF+rfcPurxrMS8OHOatztf9QJZ37UvXboVRg0CkgBPuSjnJnmDl51jZeoHk5qRwknWuyb6yUdzt7tABpk2DG26Au+/2Lgbyt795g/EZKojAvgY42sza4aViRgLqjueovfKszjGoYiUnrFjEsasWU7LuI9rurmFnXj6Lew3kz98+j4VFg3ivaFDY5VOjDVL6yecHVhURg99KjVy+QHJTyTzXZD6vzWp3Xp43gal7d7j+eq/iato0aN8+KW1MVMKB3Tm3wMyeBt4DaoDFwKRE9yuZacyQIti9m9kPP8tR773GqE/epnirN8C5rFs/Hj/qLP7Vfwjn/9dF3PX6mvoeV6udNRBmOdtw65SHRKuOCD2eqmVm/VZqZPMFkuP99JOt55pQu8eN85YeGDvWG0ydNcvr0WcYzTwVf5yDd9+FyZO98q9Nm9iVl89bfY9k9oBjmHvQUCr29dbaCJernL64rFHuvaFIuc1o+fywF4ZIYm49nlmdqUwPBaW5ee10nmsix0643U8/DRdeCMccAy+9BPvu/Wk0GbQeuwRjzRovnzh5MnzyiXdl+bPOgnPP5fmiwYx7eZXvYNBv/AthDxGpPDHSIFe4S8FBcgcos3VQ1K90n1+8gTYjqo+eesqbnXrssfDCCylJy/gN7Em4Cq1kPedgzhwYM8a7YPAtt0CPHvDww7Bxo1fTe+GFnDn8EO489wiKCgswvCAQ7R+rKEIOM1Juc9yogRTkN64+KMjPCxvUIbkDlJHakulpB7/SOegbCtJllVU49oxfTF9cFvF3bpuxNGqaLiUuuMDr9Lz5pve/snNn7N9JEa0Vk8UC/xi8fTs8+ig8+CAsX87Ozl148rgLeXjgCGr79efELt149aFFEStUYok3txmpOiLSao/JHKBMdqVGuqVz0DfemabhroUakvLqo4su8haT+8EPvNsTT/ia3ZxsCuxZKtAldysrvQWQ7rsPNm+Gb3+bhb+9nyu392Obq+ulVlbx9/lr6n+lOcdrTnCM9MaRjkG7VFXgBCmdF/b2K95PC9F65WmpPrr8cigv98ohe/XySiLTTIE9SwWynsbmzd6L8MEHvZUQzzgDbrwRhg3j2gnz2Oai936as35HUMFxn/xW9edfWJDPbaMPy7qgm2zxLqYF6flEEu+nhWi98rSlxsaNg7IyuOceKCry1ntPIwX2LBXpxV1WWRV73Y4dO7xV6yZM8GbVXXCBF9AHD465f7/tSJZwg2Y7a2rDbperaRO/4n3zT9cnkng/LfhZDjnlzODee72loK+7DgYM8IoM0iT9ySAJa/riMoZNmEf/8S8wbMK8vQaSon3kjDgAtXs3PPKI96K76Sbvuo8ffghTpjQK6rH23/RY4dqXLLFq26F5g3HZLtzrJVtmwY4ZUhTXIHykgeyGyyGnRatW8Pjj3jVUL77YW18mXU1J25ElIj+BKdyLu6lGAe/dd+Hoo+Hqq70lbN98E557zlubJQw/+w9JZeD0E6z8BP9cEun1UtguP+z2mTgLdsyQIt4aP4KVE87grfEjYo67xPNGkFIFBTB9ulfXPnq0l+5MA6ViMpCfj9BNc6KRZiNs37DJu1jvpEleyeI//uFNrIixQl24nOuJh3Tj1Y8rwn4MTtV62X7ysdnSU20okdRRpNdL29atwk7kyoUSzYweyO7dG559Fo4/Hr73PW92auvUhloF9gwULX/eUMMXd7gJJqd88jYT5jxIbdWXTP3Oudzxre/RcXUXxpWu9/VPEemfJ9IszFQETj/52GxbryXRCqdIf/etVdXc+73BLX6sIS2OPhr+9Ce48kr4zW+8WwopsGegSIEp2roqDQNep6ovue2VP3POR6+xvv8hnDfydhZ36QfAl4mURcZoXyoCZ6qvSpQKiVY4RXs+Mrpnm+uuuMJLed5+uzc7NYUX6lCOPQONGzWQcIkSR+Qa3lDe8ZxNS5nzyE856+M3WfajX3HhVffXB/WQRPPN6Z6FGSsfm9E52DASTR2l+/mQKB54wBvHuvRSWJ+6C8upx56Bxgwp4r+mlIZ9LOI/e00NY556kDF/vdO7yvrf5jJoyBDWRlifJZG0STbMwsymnmqin4Cy4flosdq189aUKSnxZqnOm5eSi3QosGeoIh//7KEBN7dmDQ+9eDffWP2hV/Vy//3Qrh3TF5fRKsKCWYmmTbIpcGa6IFJHej4y2KBB8NBD3gzVCRPg5puTfkilYjJUrI/XoQG3PksW8Pxj13Lghs+4bsz1TP/pbfVB/cZpH4QN6sn6mB6r9l7Cy7bUkTTD97/v9dhvvRXeeSfph1OPPUPF+ng9cfZyzn1nJre98mdWF/Zk7Lm3sKJLb96uG3ALNyAH3pK3yQgaga5d0wKpx53jzOCPf4Svv4bOnZN/uCDWYzezQuBh4HC8Mb4rnXNvR9q+Ja/HHq5eGaLnR5v+zvUjDmDbj3/G9xe/wLwDSrh29Di+bOutBR1a2zyeC0MEId3reYu0BKm8mDXA/cAs59z5ZtYGaBfQfnNKuF7tuKfeB4Pq3a7+voY93aa/U1m+mS4XjePsFe/x56Hnctfxl1Pbak/KJpQ7T3VJYjZOChLJVQnn2M2sI3Ac8AiAc26Xc64y0f3monDpkepaVx/UQ5pe1zP0O/vt2Mo/nryZo1eWcsNpv+DOE69sFNQb5s5TXQIX6Q0jUycFieSyIAZPDwAqgL+a2WIze9jMMvPS3WkWT+81tG3oa++t5Tz1xPUM/Hw1Pzr3ZqYc2XiyQ2FBfqPceaoH5FRLLZI5gkjFtAaOAn7unFtgZvcD44H/abiRmY0FxgIUFxcHcNjsEyk9Emnb0NeCT5fz9ym3UFC9k0u+dzuLeu+9cFf7tq3DTtRJ1YCcaqlFMkcQgX0dsM45t6Du56fxAnsjzrlJwCTwBk8DOG7WCVevnN/KGuXYoXFP9zcDjG/cfhO1GBdcchefdOsXdt9NPw2kYz3yVL2RaK11kegSDuzOuY1mttbMBjrnlgMjgY8Sb1ruidSrDXffmCFF8NFHjLzmIr5s05pzzr+DT/eLHLyaTlzK1dLDXD43kaAEVe44GK/csQ2wArjCObcl0vYtudzRt2XL4MQT+Xq349zzb+ejTr0iblqQn9cof57LpYe5fG4isaS03NE5VwrEPJj49NlnMGIEmHHF9yfwUZtue22SZ0atc2FTEblcepjL5yYSFM08zTTl5TBqFOzaxdxJT/P2uzvCblbrXMSJRtm2Hnk8cvncRIKitWLSIOKaKtu2wWmnwYYNvH7vY/ysdGfEfUQLZLlcepjL5yYSFPXYUyzS4F+rXTsZfdMPYckSmDmTm95vS1V1+PRCrECWy6WHuXxuIkFRYE+xsFfL2VXDPj/6Ibw/DyZPhtNOY/3r4ddRB3xNNMrlRaVy+dxEgqBUTIqFG+S75u2pnPL+PPi///OW9yRyqqWo7nJnIiKRKLCnQMOceitrfNG7UZ/8m3Fv/o1Zg0+C8XvmdSmXLCLNpVRMkjXNqTe88MWh5Su49/m7eb9oILse+pO3ZnMd5ZJFpLkU2JMs0gUv9t+xlb9M+y1fFXRg/aP/YPTRB+61jXLJItIcCuxJFi6n3qp2N/fMmEjRri/hX//itG8elYaWiUiuUmBPsnATan7+7ykcu7oUHn4YvvnN+vsbLm5V2C4f52BrVbXSMCISFw2eJlnTQdBhq0q59q1/subM8+HKK+vvD+XiyyqrcMCWHdVUVlXj2FPrrotDi4gfCuxJ1vCCF92/3MwDz/+erw4YQPGTjzUaLI2Uiw9peFUlEZFolIpJkJ+1wccMKWLMkT28hb3cLpj5LLRvfJEpP4tYaaErEfFDgT0Bca0NPnEivPEGPPYYHLr3FZD8XF1JC12JiB9KxSQg7PIA1bv535lLG29YWgq//jVccAFcdlnYfYWbkBRuGxGRWBTYExApNbJlR/Wegc6vv4ZLL4WuXXnhJ79m2F2v7r2qI3ty8XlNZqaGFBbkqypGRHwJLLCbWZ6ZLTaz54PaZ6aLlhqpH+i85RZYupR/3/Q7rpu3rr7qJVyly5ghRdz93W+EXUrgttGHJeMURCQHBdljvxZYFuD+Ml601Mj6yip47TW45x74yU8Y91WvsGmbppUuDatoDG/RLz+rOYqIhAQyeGpmvYEzgDuAXwWxz2wwZkgRt81YSmVV9V6P9W9nXp36gQfCxIms/+1rYfcRLp2jpQREJBFB9djvA64HagPaX9a4bfRhYVMnf/50Bqxc6c0ubd8+YtrGwV75dhGRRCQc2M3sTGCTc25RjO3GmtlCM1tYUVGR6GEzRrjUyR8HwYAn/gJXXw3HHw9Er3rRzFIRCZK5BsvINmsHZncC3wdqgH2AjsA059ylkX6npKTELVy4MKHjZqyaGhg6FNavh2XLoHPn+odCk5ki1asXFRbw1vgRqWqpiGQZM1vknCuJtV3CPXbn3I3Oud7OuX7AhcC8aEE95913HyxeDP/v/zUK6uD17t8aP4LwBY2aWSoiwVAde5BWrvQmIo0eDeefH3GzSPl2zSwVkSAEGtidc685584Mcp9Z5Ze/pAbjnEEX0f/GFyMOiuqydyKSTForJiizZ8Nzz3H/iVewmA5A5LVjdNk7EUmmhAdPmyPnBk937YIjj2RdxZeMuOJBdrXOb/SwBkVFJAh+B0/VY4/Bz7K8PPAALF/Oref9eq+gDhoUFZHUUmCPwteyvBs3wm23wemn83HJ8RAmiGtQVERSSVUxUURalrfR+i433eSt4HjvvRoUFZGMoB57FJFSKPX3l5Z6F8647jo4+GDG1D2uQVERSScF9igiXdWoPrUyfrw3Cemmm+of0wJeIpJuSsVEETW1MneuV+J4881QWJimFoqI7E099igi1pt/oycMPRv69oVrrklzK0VEGlNgjyFsamXKFFi0CCZPhrZt09MwEZEIFNh9CtWzV2zexquP/pJ9Bwyi08UXp7tZIiJ7UY7dh1A9e1llFReWzqLoiw1cV3Ix05dsTHfTRET2osAew/TFZfz31Pepqt5N2+qdXDP/Keb3OZw5fQbvdb1SEZFMoMAeRainvrtuPZ2L3p9N96++4N5jLwEzLRUgIhlJgT2KhjNP21bv5CcLnubfxUeyoPgIQEsFiEhmUmCPomGPPNRbv3/YRYCWChCRzBXExaz7mNmrZrbMzJaa2bVBNCwThHrkod7628VHsKD4CPLMuPPcIzTDVEQyUhA99hrgv51zg4CjgWvM7NAA9pt240YNJL+VNeqt57cy7v7uNxTURSRjBXEx6w3Ouffqvv8SWAbkTNRrW7Ontz6/+EgiXolaRCRDBJpjN7MEWhNeAAAMfklEQVR+wBBgQZD7TZeJs5dzTunLjXLr1budyhxFJKMFFtjNbF/gGeC/nHPbwjw+1swWmtnCioqKoA6bVOVffMUP33mWRb0OYX6fI+rvV5mjiGSyQAK7meXjBfUnnHPTwm3jnJvknCtxzpV069YtiMMm3cVr36F4azl//vZ5YHtyMCpzFJFMFkRVjAGPAMucc/ck3qQM4Ry/Kn2OFV16M2fAt+vvVpmjiGS6IHrsw4DvAyPMrLTudnoA+02vuXMp/PhDtl5zLb06t8eAosIClTmKSMZLeHVH59y/yMVakbvugh49GHLTz3mrydK8oZUedfk7EclEWrY3nPfeg1degQkT9lpvPbR+TGipgbLKKm6c9gGAgruIZAQtKRDOxInQoQP8+Md7P9Rg/ZiQqurdKoEUkYyhwN7E7BffoWbqVCYNOplhDy1i+uKyRo9HKnVUCaSIZAoF9gamLy5jzR2/xxw8ftSZ9WmWhsE9UqmjSiBFJFMosDdw//NLOP+9Wbw84GjKOu0P7J1mGTdqIAX5eY1+TyWQIpJJNHjawNB/z6Lz11/y2DfPanR/wzRLaIBUVTEikqkU2EOc4+rS51nWrR8L+hze6KGmaZYxQ4oUyEUkYykVE/LGGwzYuILJ3xrdaPmA/FamNIuIZBUF9pA//IFdnQqZedgJje/PvalXIpLjFNgBVq+G6dOZMvhUvmrVptFDWqZXRLKNAjvAQw8B8KdDTwn7sGrURSSbKLDv2gWPPgqjR0Nx37CbqEZdRLKJAvtzz0FFBfzoR6pRF5GcoHLHSZOgb184+WTG5HlBXTXqIpLNWnZg/+wzbxXH3/4W6oK6atRFJNu17FTMX/7iBfQrrkh3S0REAtNyA/uuXfDXv8KZZ0KReugikjuCupj1qWa23Mw+NbPxQewz6WbMgE2bYOzYdLdERCRQQVzMOg94EDgNOBS4yMwOTXS/STdpEvTpA6NGpbslIiKBCqLHPhT41Dm3wjm3C3gSODuA/SbPihUwZw5cfXX9oKmISK4IIrAXAWsb/Lyu7r5GzGysmS00s4UVFRUBHDYBjz/uLfSlQVMRyUFBBPZwy2S5ve5wbpJzrsQ5V9KtW7cADttMtbUweTKMHOmlYkREckwQgX0d0DBC9gbWB7Df5HjzTVi1Ci6/PN0tERFJiiAC+7vAADPrb2ZtgAuBGQHsNzkefxz23RfOOSfdLRERSYqEZ54652rM7GfAbCAPeNQ5tzThliXD9u3w1FNwwQXQvn26WyMikhSBLCngnHsReDGIfSXVs8/CV18pDSMiOa1lzTx9/HHo1w+GD093S0REkqblBPa1a2HuXLjsMmjVck5bRFqelhPhnngCnPMCu4hIDmsZgd05Lw0zfDgceGC6WyMiklQtI7CXlsLHH8Oll6a7JSIiSdcyAvuTT0Lr1nDeeeluiYhI0uV+YK+t9QL7qFHQpUu6WyMiknS5H9jnz4c1a+DCC9PdEhGRlMj9wP7Pf8I++8DZmb2SsIhIUHI7sNfUwNSp3uXvOnRId2tERFIitwP7a695l79TGkZEWpDcDuxPPun11E8/Pd0tERFJmdwN7Dt3wjPPwJgxUFCQ7taIiKRM7gb2l1+Gykq46KJ0t0REJKVyN7BPmQL77QcnnZTuloiIpFRuBvadO2HmTC8Nk5+f7taIiKRUQoHdzCaa2cdmtsTMnjWzwqAalpC5c2HbNi0hICItUqI99jnA4c65I4FPgBsTb1IApk2Djh1h5Mh0t0REJOUSCuzOuZedczV1P84HeifepATV1MD06d6kpLZt090aEZGUCzLHfiXwUoD7a5433oDNm5WGEZEWK+bFrM3sFaBHmIduds49V7fNzUAN8ESU/YwFxgIUFxc3q7G+TJvm1a2PGpW8Y4iIZLCYgd05F7Ve0MwuB84ERjrnXJT9TAImAZSUlETcLiG1tV5gP+00aN8+KYcQEcl0MQN7NGZ2KnADcLxzbkcwTUrA/PmwYYPSMCLSoiWaY38A6ADMMbNSM/tTAG1qvmnTvLr1M85IazNERNIpoR67c+6goBqSMOe8tWFOPhk6dUp3a0RE0iZ3Zp6WlsKqVUrDiEiLlzuBfcYMMIOzzkp3S0RE0iq3Avsxx0C3buluiYhIWuVGYC8rg/feU29dRIRcCezPP+99HT06ve0QEckAuRHYZ86EAw6AQYPS3RIRkbTL/sC+fTu88oqXhjFLd2tERNIu+wP7K694F9ZQGkZEBMiFwD5zpjchafjwdLdERCQjZHdgr631Bk5PPVWXwBMRqZPdgf3dd6G8XGWOIiINZHdgnzkT8vK8ZXpFRATIhcB+7LGw337pbomISMbI3sC+bh0sWaIlekVEmsjewD57tvdVaRgRkUayN7DPmgVFRXDYYeluiYhIRgkksJvZdWbmzKxrEPuLqaYG5szxeuuabSoi0kjCgd3M+gAnA2sSb45P8+fD1q1e/bqIiDQSRI/9XuB6wAWwL39eeskrcxw5MmWHFBHJFgkFdjMbDZQ5594PqD3+zJoF3/kOFBam9LAiItkg5sWszewVoEeYh24GbgJO8XMgMxsLjAUoLi6Oo4lNlJd7F9W4447m70NEJIfFDOzOuZPC3W9mRwD9gffNG8DsDbxnZkOdcxvD7GcSMAmgpKSk+WmbUJmj8usiImHFDOyROOc+APYP/Wxmq4AS59znAbQrslmzYP/9YfDgpB5GRCRbZVcd++7d8PLLXm+9VXY1XUQkVZrdY2/KOdcvqH1FtHAhbN6sNIyISBTZ1e2dNcubkHTyyeluiYhIxsquwN67N1xxBXRNzQRXEZFslF2B/aqr4JFH0t0KEZGMll2BXUREYlJgFxHJMQrsIiI5RoFdRCTHKLCLiOQYBXYRkRyjwC4ikmMU2EVEcow5l7oLH9Uf1KwCWJ3yA8enK5DclSpTI1fOA3QumShXzgOy41z6Oue6xdooLYE9G5jZQudcSbrbkahcOQ/QuWSiXDkPyK1zUSpGRCTHKLCLiOQYBfbIJqW7AQHJlfMAnUsmypXzgBw6F+XYRURyjHrsIiI5pkUHdjM71cyWm9mnZjY+zONtzWxK3eMLzKxf6lvpj49z+ZWZfWRmS8xsrpn1TUc7/Yh1Lg22O9/MnJllZCWDn/Mws+/WPS9LzewfqW6jXz5eX8Vm9qqZLa57jZ2ejnbGYmaPmtkmM/swwuNmZn+oO88lZnZUqtsYCOdci7wBecBnwAFAG+B94NAm2/wU+FPd9xcCU9Ld7gTO5USgXd33P8nmc6nbrgPwBjAfKEl3u5v5nAwAFgOd637eP93tTuBcJgE/qfv+UGBVutsd4VyOA44CPozw+OnAS4ABRwML0t3m5txaco99KPCpc26Fc24X8CRwdpNtzgYer/v+aWCkmVkK2+hXzHNxzr3qnNtR9+N8oHeK2+iXn+cF4LfA74CvU9m4OPg5jx8CDzrntgA45zaluI1++TkXB3Ss+74TsD6F7fPNOfcG8EWUTc4GJjvPfKDQzHqmpnXBacmBvQhY2+DndXX3hd3GOVcDbAW6pKR18fFzLg1dhdcryUQxz8XMhgB9nHPPp7JhcfLznBwMHGxmb5nZfDM7NWWti4+fc7kNuNTM1gEvAj9PTdMCF+//UkZqne4GpFG4nnfTEiE/22QC3+00s0uBEuD4pLao+aKei5m1Au4FfpCqBjWTn+ekNV465gS8T1BvmtnhzrnKJLctXn7O5SLgMefc3WZ2DPC3unOpTX7zApUt//NRteQe+zqgT4Ofe7P3x8f6bcysNd5HzGgf49LFz7lgZicBNwOjnXM7U9S2eMU6lw7A4cBrZrYKLw86IwMHUP2+vp5zzlU751YCy/ECfabxcy5XAVMBnHNvA/vgrb2SbXz9L2W6lhzY3wUGmFl/M2uDNzg6o8k2M4DL674/H5jn6kZYMkzMc6lLX/wZL6hnai4XYpyLc26rc66rc66fc64f3njBaOfcwvQ0NyI/r6/peIPamFlXvNTMipS20h8/57IGGAlgZoPwAntFSlsZjBnAZXXVMUcDW51zG9LdqLile/Q2nTe8EfBP8Eb8b6677zd4gQK8F+dTwKfAO8AB6W5zAufyClAOlNbdZqS7zc09lybbvkYGVsX4fE4MuAf4CPgAuDDdbU7gXA4F3sKrmCkFTkl3myOcxz+BDUA1Xu/8KuDHwI8bPCcP1p3nB5n62op108xTEZEc05JTMSIiOUmBXUQkxyiwi4jkGAV2EZEco8AuIpJjFNhFRHKMAruISI5RYBcRyTH/HxY877DR7oizAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f457f58fac8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = PolynomialRegression(4).fit(X, y)\n",
    "plt.scatter(X, y)\n",
    "plt.plot(X_test, model.predict(X_test),c='r');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detecting Data Sufficiency with Learning Curves\n",
    "\n",
    "As you might guess, the exact turning-point of the tradeoff between bias and variance is highly dependent on the number of training points used.  Here we'll illustrate the use of *learning curves*, which display this property.\n",
    "\n",
    "The idea is to plot the mean-squared-error for the training and test set as a function of *Number of Training Points*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.learning_curve import learning_curve\n",
    "\n",
    "def plot_learning_curve(degree=3):\n",
    "    train_sizes = np.linspace(0.05, 1, 20)\n",
    "    N_train, val_train, val_test = learning_curve(PolynomialRegression(degree),\n",
    "                                                  X, y, train_sizes, cv=5,\n",
    "                                                  scoring=rms_error)\n",
    "    plot_with_err(N_train, val_train, label='training scores')\n",
    "    plot_with_err(N_train, val_test, label='validation scores')\n",
    "    plt.xlabel('Training Set Size'); plt.ylabel('rms error')\n",
    "    plt.ylim(0, 3)\n",
    "    plt.xlim(5, 80)\n",
    "    plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what the learning curves look like for a linear model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XmcXFWZ+P/Pc6uqu3pPZw8JkCAJS/YQFmVfZMBREQQJ6kgYlX2U73dkhpn5DggyP535OupXRZig4MawL6KyCcIgikgSQgybSUgCSSB7b+mtqu7z++Pcqq6uvr2mqqu687xfr8pd6ta9pzvV97nnnHueK6qKMcYYk8srdgGMMcaUJgsQxhhjQlmAMMYYE8oChDHGmFAWIIwxxoSyAGGMMSZUwQKEiMRF5E8i8qqIvCYiN4ZsUy4i94rIOhF5SUSmF6o8xhhjBqeQNYgO4DRVnQ8sAM4SkeNytvk8sEdVDwW+Dfx7ActjjDFmEAoWINRpCRZjwSt3VN45wE+C+QeA00VEClUmY4wxAxct5M5FJAKsAA4FblHVl3I2mQq8C6CqSRFpBMYBO3P2cylwKUBVVdVRhx9+eCGLbYwxo86KFSt2quqEwXymoAFCVVPAAhEZAzwsInNUdU3WJmG1hR65P1R1GbAMYPH8I3X5L++AaDlEyoJpOUTLuqbGGGO6EZFNg/1MQQNEmqo2iMhzwFlAdoDYDBwIbBaRKFAH7O5/hylItLpXD9JH8CgHa8EyxpgBKViAEJEJQCIIDhXAGfTshH4UuBh4ETgf+K3uc/ZAhWS7e3WEvB0JgsWYg6y2YYwxfShkDWIK8JOgH8ID7lPVX4nITcByVX0U+BHwMxFZh6s5LClgeZxUp3u17YGaSQU/nDHGjFQFCxCquhpYGLL++qz5duCCQpWhTxYgzH4qkUiwefNm2tvbi10UUwDxeJxp06YRi8X2eV/D0gdRkpJtkGiDWEWxS2LMsNq8eTM1NTVMnz4du6t8dFFVdu3axebNm5kxY8Y+72//TrXRtqfYJTBm2LW3tzNu3DgLDqOQiDBu3Li81Q4tQBizH7LgMHrl8/92/w4QqU7oaOl/O2OM2Q/t3wECoK3/YRfGmPxpaGjgBz/4wZA++5GPfISGhoY+t7n++ut5+umnh7R/050FiLYG2NehF8aYAesrQKRSqT4/+9hjjzFmzJg+t7nppps444wzhly+QunvZytFFiA0Be2NxS6FMfuN6667jvXr17NgwQKuvfZannvuOU499VQ+/elPM3fuXAA+8YlPcNRRRzF79myWLVuW+ez06dPZuXMnGzdu5IgjjuCLX/wis2fP5swzz6StrQ2ApUuX8sADD2S2v+GGG1i0aBFz587lzTffBGDHjh18+MMfZtGiRVx22WUcfPDB7NzZLQUcqVSKpUuXMmfOHObOncu3v/1tANatW8cZZ5zB/PnzWbRoEevXr0dVufbaazPb3nvvvQChP9vPf/5zjjnmGBYsWMBll11GKpXq9VjFtv/e5pqtbQ9U9H1VYsxodOMvX+P1rU153eeRB9Ryw8dm9/r+N77xDdasWcOqVasAdxL905/+xJo1azK3Zt5xxx2MHTuWtrY2jj76aD75yU8ybty4bvtZu3Ytd999N7fffjuf+tSnePDBB/nsZz/b43jjx49n5cqV/OAHP+Cb3/wmP/zhD7nxxhs57bTT+Kd/+ieeeOKJbkEobdWqVWzZsoU1a1x2oHTT1mc+8xmuu+46zj33XNrb2/F9n4ceeohVq1bx6quvsnPnTo4++mhOOukkgG4/2xtvvMG9997L73//e2KxGFdeeSV33XUXs2fPDj1WsVkNAqCjCfyRV/0zZrQ45phjut23/93vfpf58+dz3HHH8e6777J27doen5kxYwYLFiwA4KijjmLjxo2h+z7vvPN6bPPCCy+wZIlL3HDWWWdRX1/f43OHHHIIb7/9Nn/3d3/HE088QW1tLc3NzWzZsoVzzz0XcIPSKisreeGFF7jooouIRCJMmjSJk08+mZdffrnHz/bMM8+wYsUKjj76aBYsWMAzzzzD22+/HXqsUmA1CAD1XTNT5dhil8SYYdXXlf5wqqqqysw/99xzPP3007z44otUVlZyyimnhN7XX15enpmPRCKZJqbetotEIiSTScANKOtPfX09r776Kk8++SS33HIL9913H9/5zndCt+1rf9k/m6py8cUX8/Wvf73HdrnHuuOOO/otY6FZDSLNxkQYMyxqampobm7u9f3Gxkbq6+uprKzkzTff5I9//GPey3DCCSdw3333AfDUU0+xZ0/Pv/+dO3fi+z6f/OQn+drXvsbKlSupra1l2rRpPPLIIwB0dHTQ2trKSSedxL333ksqlWLHjh08//zzHHPMMT32efrpp/PAAw+wfft2AHbv3s2mTZtCj1UKrAaR1tEEqQRE9j1/iTGmd+PGjeP4449nzpw5nH322fz1X/91t/fPOussbrvtNubNm8dhhx3GccflPql4391www1cdNFF3HvvvZx88slMmTKFmpqabtts2bKFSy65BN/3ATJX/T/72c+47LLLuP7664nFYtx///2ce+65vPjii8yfPx8R4T/+4z+YPHlyplM87cgjj+Tmm2/mzDPPxPd9YrEYt9xyCxUVFaHHKjbZ5+zaw2zx/CN1+eN3FWbntVOhemJh9m1MiXjjjTc44ogjil2Mouro6CASiRCNRnnxxRe54oorMp3mo0HY/7GIrFDVxYPZj9UgsrXtsQBhzH7gnXfe4VOf+hS+71NWVsbtt99e7CKVJAsQ2RKtkGiHWLzYJTHGFNDMmTN55ZVXil2Mkmed1Lmss9oYYwALED1ZgDDGGMACRE+pDujcW+xSGGNM0VmACGO1CGOMsQARqm2PZXg1poRUV1cDsHXrVs4///zQbU455RSWL1/e536+853v0NramlkeSPrw/ZkFiDB+0g2cM8aUlAMOOCCTqXUocgPEQNKHF0OppAa3ANEba2YypiD+8R//sdvzIL761a/yn//5n7S0tHD66adnUnP/4he/6PHZjRs3MmfOHADa2tpYsmQJ8+bN48ILL+yWi+mKK65g8eLFzJ49mxtuuAFwCQC3bt3Kqaeeyqmnngp0pQ8H+Na3vsWcOXOYM2dOJudSX2nFs91///3MmTOH+fPnZ7K4plIpvvKVrzB37lzmzZvH9773PcAl7Fu4cCFz587lb//2b+no6MiU5aabbuKEE07g/vvvZ/369Zx11lkcddRRnHjiiZlR2WHHKhQbB9Gb9kbwffAshppR7PHr4P0/53efk+fC2d/o9e0lS5ZwzTXXcOWVVwJw33338cQTTxCPx3n44Yepra1l586dHHfccXz84x/v9RnLt956K5WVlaxevZrVq1ezaNGizHv/9m//xtixY0mlUpx++umsXr2aL33pS3zrW9/i2WefZfz48d32tWLFCu68805eeuklVJVjjz2Wk08+mfr6+gGlFb/pppt48sknmTp1aqbJatmyZWzYsIFXXnmFaDTK7t27aW9vZ+nSpTzzzDPMmjWLz33uc9x6661cc801gMsO+8ILLwAub9Ntt93GzJkzeemll7jyyiv57W9/G3qsQrGzX2/Uh3ZrmzQm3xYuXMj27dvZunUrr776KvX19Rx00EGoKv/8z//MvHnzOOOMM9iyZQvbtm3rdT/PP/985kQ9b9485s2bl3nvvvvuY9GiRSxcuJDXXnuN119/vc8yvfDCC5x77rlUVVVRXV3Neeedx+9+9ztgYGnFjz/+eJYuXcrtt9+eaR56+umnufzyy4lG3XX42LFjeeutt5gxYwazZs0C4OKLL+b555/P7OfCCy8EoKWlhT/84Q9ccMEFmQcLvffee70eq1CsBtGXtj2WAtyMbn1c6RfS+eefzwMPPMD777+feS7DXXfdxY4dO1ixYgWxWIzp06eHpvnOFla72LBhA9/85jd5+eWXqa+vZ+nSpf3up6+cdANJK37bbbfx0ksv8etf/5oFCxawatUqVLVH+frLfZdODe77PmPGjAnNDxV2rNyHKeWL1SD60tEMqWSxS2HMqLNkyRLuueceHnjggcxdSY2NjUycOJFYLMazzz7Lpk2b+tzHSSedxF13ucSda9asYfXq1QA0NTVRVVVFXV0d27Zt4/HHH898prdU4yeddBKPPPIIra2t7N27l4cffpgTTzxxwD/P+vXrOfbYY7npppsYP3487777LmeeeSa33XZb5hkUu3fv5vDDD2fjxo2sW7cOcJlhTz755B77q62tZcaMGdx///2ACyyvvvpqr8cqFKtB9EldM1PV+P43NcYM2OzZs2lubmbq1KlMmTIFcI/y/NjHPsbixYtZsGABhx9+eJ/7uOKKK7jkkkuYN28eCxYsyDx/Yf78+SxcuJDZs2dzyCGHcPzxx2c+c+mll3L22WczZcoUnn322cz6RYsWsXTp0sw+vvCFL7Bw4cJen1KX69prr2Xt2rWoKqeffjrz589nzpw5/OUvf2HevHnEYjG++MUvcvXVV3PnnXdywQUXkEwmOfroo7n88stD93nXXXdxxRVXcPPNN5NIJFiyZAnz588PPVahFCzdt4gcCPwUmAz4wDJV/X8525wC/ALYEKx6SFVv6mu/BU33HSZWBRNmDd/xjCkwS/c9+o2EdN9J4O9VdaWI1AArROQ3qprbW/Q7Vf1oAcuxbxJ7IdkB0fL+tzXGmFGkYH0Qqvqeqq4M5puBN4CphTpeQdmYCGPMfmhYOqlFZDqwEHgp5O0PisirIvK4iJTGE9RzWYAwo8xIe5KkGbh8/t8WPECISDXwIHCNqubmr1gJHKyq84HvAY/0so9LRWS5iCzfsasIJ+tkO3S29r+dMSNAPB5n165dFiRGIVVl165dxOP5eehZQZ9JLSIx4FfAk6r6rQFsvxFYrKo7e9tm2Dup06omQt3IbCEzJlsikWDz5s39jg0wI1M8HmfatGnEYrFu60uqk1rcCJEfAW/0FhxEZDKwTVVVRI7B1Wh29bnjlu3Qsg2qJ+W7yH1r22MBwowKsViMGTNmFLsYZgQo5F1MxwN/A/xZRNLDAf8ZOAhAVW8DzgeuEJEk0AYs0f6qNO0NcM9nYNZfwYLPQO0BBfsBuvETbuBcec3wHM8YY4qsoE1MhbB4zixdfv0H4a1fg5+CQz8MCz8DYw4q/MErxkL9wYU/jjHG5FlJNTEVTCQKJ1wDi/4GXr0X3ngU1j4Fh5zi1o09pHDHtgyvxpj9yMgLEGmV4+CDV8KCT8Of74PXHoa3n4XpJ8Kiz8H4mfk/pqagoxEq6vO/b2OMKTEjN0CkVYyBYy6FeUtgzYOw5gHY+Ds46DgXKCYemd/jte2xAGGM2S+MnraSeC0svgQ+fS8s/jxsex0euRJ+/RV4b3X+jtPe5Po+jDFmlBs9ASKtrNr1RXz6Hjj2cti9Hn75Jfjll2HLCtjnTnmFNnuQkDFm9Bt9ASItVgnzl8BFd8MHr4bGLfDrv4dHr4Z3/rhvgcJSbxhj9gOjN0CkReMw93xYchec8L9g70544jp45sah77OzGZKd+SujMcaUoNEfINKi5XDkOXDhz2HehfD2c/v2sHarRRhjRrn9J0CkRWJw1FKIj4EVPxn6fixAGGNGuf0vQADEKlz/xJblsO21oe0j2QaJng8vN8aY0WL/DBDgmpvidbDix0Pfh9UijDGj2P4bIGIVri9i88uwPfcpqANkAcIYM4rtvwECYPYnoLx26H0RqU7oaMlvmYwxpkTs3wEiVulqEe++BNvfGNo+rBZhjBml9u8AATD7XFeLWDnEWkR7Qx5GZxtjTOmxAFFWCXMvcKOrd7w5+M/7SZcG3BhjRhkLEABzznNPilvx06F93pqZjDGjkAUIgLKqoBbxB9j5l8F/vsMyvBpjRh8LEGlzznOZYIdyR5P61sxkjBl1LECklVW7pH6bfg871w7+89bMZIwZZSxAZJvzSdfctHIIfREdzZBK5L9MxhhTJBYgspXXwJzz3SNLd60f5IftQULGmNHFAkSuuedDrGpo4yLadue/PMYYUyQWIHKV17gO6w3Pw+63B/fZRCsk2gtTLmOMGWbRYhegJM29ANY86Poizvjq4D67ay1IVtztMco6ZNR16EjsnHXiwfhZ7sFHxhgzDKwGESZeC7PPg7f/B3ZvGNxn/aRL4pd++YmcV7LnS1MhL7/7y0+6svh+YX5mY4zJYQGiN/MugFh8aHc0FUqyDRo2FrsUxpj9hAWI3sTrXCK/t5+DPRuLXZou7Y3Q/H6xS2GM2Q8ULECIyIEi8qyIvCEir4nIl0O2ERH5roisE5HVIrKoUOUZknmfcm3+K39W7JJ01/yejdw2xhRcIWsQSeDvVfUI4DjgKhE5Mmebs4GZwetS4NYClmfw4mNcLWL9b6FhU7FL092eTXbHlDGmoAoWIFT1PVVdGcw3A28AU3M2Owf4qTp/BMaIyJRClWlI5l1YmrUITcGeDZYk0BhTMMPSByEi04GFwEs5b00F3s1a3kzPIIKIXCoiy0Vk+Y5dw5zzqGIMHHlOUIt4t//th1OyvbT6R4wxo0rBA4SIVAMPAteoalPu2yEf6TEoQFWXqepiVV08YVx9IYrZt3kXgheDV0qsFgEu1XjTe8UuhTFmFCpogBCRGC443KWqD4Vsshk4MGt5GrC1kGUaksqxrhax7mlo3Fzs0vTU8r5lkzXG5F0h72IS4EfAG6r6rV42exT4XHA303FAo6qW5uXw/AvBi5ZmLQKg4R1ItBW7FMaYUaSQNYjjgb8BThORVcHrIyJyuYhcHmzzGPA2sA64HbiygOXZN5Xj4IiPw9rfQNOWYpemJ/WDkdbWaW2MyY+C5WJS1RcI72PI3kaBqwpVhrxbcBG88Sis/Dmc8o/FLk1PqQ7XaT32EJA+f/XGGNOvkTeS2itifsHKcXDEx2Dtk9BUel0lQNBpXaJlM8aMKCMvQETKoHYa/VROCmf+ReBF4JWfF+f4A7F3O7TasymMMftm5AUIgOoJQTNKZPiPXTUeDv8o/OVJl/KiVDW+C52txS6FMWYEG5kBAlxK7vGzIFKE5yPMv8g9n+GVu4b/2AOlvhtpnUoWuyTGmBFq5AYIcOm4x8+CsurhPW71RDj8r+Gtx0s7s2qq03Vahz6QyBhj+jayAwRAJArjDoWKscN73AWfdncKrSrhWgRAZ3Np3pZrjCl5Iz9AgDtR1x8MtT3SOBVOdi2iZdvwHXco9u6wTmtjzKCNjgCRVj0R6md0fyZ0IS34tJuu+u/hOd6+aHgHOvcWuxTGmBGkzzOpiERE5P8OV2HyomJM0HldVvhjVU+Cw86GNx+Dlu2FP94+UTfSOpUodkGMMSNEnwFCVVPAUUFepZEjVuGCRKyq8Mda8BlAR0Ytwk9Yp7UxZsAG0hbzCvALEfkbETkv/Sp0wfZZJBZ0Xhc4PXjNZJh1Frz5a9fWX+o6W9wYCWOM6cdAAsRYYBdwGvCx4PXRQhYqbzwP6qdDTYEfUrfwM27cwaq7C3ucfGndBXt3FrsUxpgS129iI1W9ZDgKUlA1k91jQxvecSfyvO9/Csz6K3jzl67jump8/o+Rb42bIRqH8mEeQ2KMGTH6rUGIyDQReVhEtovINhF5UESmDUfh8qqi3jU5ebHC7H/hZ12q7Vd7qUWoD37SDV5LtkOi1TX3tDdBe4O7DXXvTtfZ3fy+S7jXuNkFtT0boWFTnoObuv1ap7UxphcDSY16J/DfwAXB8meDdR8uVKEKpqwKJhwGu992J+h8qj0AZv4VrHnI9Ueo3/2VD1OPgtOvh3hdfvbnJ9ydTbUHuASE4rn8Vl7E0oUbYxDt544WEVmlqgv6WzdcFi9erMuXL9+3nfg+7F7vruDzqXUXrL4f8N2JVqTrpBs6H/LyPCBnKh40b4Pld7jHn555M4yfmd+y9yAhQSNdxt7WR93NAV7UvSzIGFMyRGSFqi4ezGcGUoPYKSKfBdJtJxfhOq1HLs9zA+p2vuWafPKlchwcd3n/2w3VAQvhN/8Kv7gKTvwKzDqzcMdCXZPYvvCirkkvEnOBJDMftWBizAgwkADxt8D3gW8DCvwhWDeyRaIuSOxaW5iO60KYeDictwyevhGe+/9cgDvuiuI+RKkvftK9kgN4VnZfwUQ83FeP7mM4VN36zLqwee2+be58RhCgMoFqX5YFomVuHI43upIVmP1Ln2cWEYkAn1TVjw9TeYZXWSWMOch11o4UFfXw19+El/4L/nw/7FoHp9/gmp5GssEEkxFDIFbp+r7KqlzW4UiJBnNjQgxkJPU5w1SW4qioh+rJxS7F4HhR+OBVcNr/ge1vwkOXwrbXil0q04NCYq97wt+eDbDtz7DtddizCfbugkR7sQtoTJ8GcjnzexH5PnAvkMn2pqorC1aq4VY7xV25tjcWuySDc+gZbiDgU/8Kv7wGjv8yHDEyxjDut1Id0NYBbUF2XS/aVbtI1zasP8aUiIHcxfRsyGpV1dMKU6S+5eUupjC+Dzv/MjKbONqb4Ldfg80vu8ehHv+l4UlWaPJPvK5AEavs6sAXD5CuPg7xuuatn8MMQN7vYhIRD7hVVe/bp5KNBJ7nnnO98619v3tnuMVr4axvwPI7YdXP3S28Z9zo0p+bkUV9d/v1UG7Bzg4ifQWU3BpKdsf+Pi9n3cKdfQt07u3RmWXJWc5+3wJfsfUZIFTVF5GrgdEfIMDdeVI/w3X89rjLpcR5ETjmCzBhFjz3dXj4MjjjqzBlfrFLZoZL+m68EfbV7Z10DxxhQltAQtb1uV3YuKSsQJv96rGt9DIfbJu+m079rvnQdfS+HcH6Huuy99XHe93u7hucgfRB/EZEvkLPPojR+Yiy8mqoOxAa3yl2SYZmxknuzqyn/hV+9b/hg1fC7POsXduMQAqaglSq2AXZbw10HATAVVnrFDgk/8UpEVXjXCqO1hGa8bR+Opx7Kzz7dfjD92DHW3Di/3bJ+YwxZoAGks11xnAUpOTUTYNkB3Q2F7skQ1NWDWd+DV75ueub2P22Wy506nNjzKgxkGyulSLyf0RkWbA8U0T6vZdSRO4IMsCu6eX9U0SkUURWBa/rB1/8AhJxV+KR8mKXZOjEg0Wfg7O+Ds3vwUOXweYC3AFmjBmVBnKbwJ1AJ/ChYHkzcPMAPvdj4Kx+tvmdqi4IXjcNYJ/DKxKFsTPcXRUj2UHHwbn/5XJFPf4P7sFG9thRY0w/BhIgPqCq/wEkAFS1jUzimd6p6vPAyO/IjlVA/cHFLsW+q5sGn7gFZpwMf/oveObG/Kc8N8aMKgMJEJ0iUkFwT5iIfADoyNPxPygir4rI4yIyu7eNRORSEVkuIst37CjCc5/jdVBzwPAfN99ile55EsdeDhueh0eudA8lMsaYEAO5i+kG4AngQBG5CzgeWJqHY68EDlbVFhH5CPAIEPqQA1VdBiwDN5I6D8cevJpJbpR1256iHD5vRGD+Evc8iadvdP0S8y5wI6/T95qnB1QhQV1Rer6XncVUPLddWTVUjnePXK0c57KxGmNGrIHcxfQbEVkJHIc7DXxZVff5/k9Vbcqaf0xEfiAi4/Ox74KpO8jd2TQammamHhWkDv8qrPhxYY4RHxMEi6ygUTXB3UZcOd7Nx2t7HwRljCmqAeUeVtVdwK/zeWARmQxsU1UVkWNwzV2l/SCidDqOHW+5x3WOdDWT4RO3ugRymRGXuGnuCE/8rvfCnq2gCh1NLktp6w433bvTjSXZu9OlMGlr6PpcmhcNAkd2IBnvgkjVBDdfMcbVcCIxCybGDKOCJacXkbuBU4DxIrIZ11QVA1DV24DzgStEJAm0AUu0v8yBpSASc3c27VzLqMhpIJK/AXTVE2Hcob2/7yehdTfs3eEez7p3Z/f5PRtcwsG+amiZJ9HFXGoUryyYxnpOI2VdgSVn6ntl+F4UJSutAlnpEiCTPkHJei9oXtPc7chqehMPr7ySaEU9UlHnRudbYDMjUMEChKpe1M/738c9qW7kKaty6SwaNhW7JCOLF3VBpL8kgp3BKPbWXS6AtDdCKhG8OrOmnTnLbqrtzWiqA5JZn/Hd+5LqRFxYGNAdGvmg4qFltWi8DuJ1SEUdEh+DBMuhr1iFpUcxRddvgAjuWtqsqh0icgowD/ipqjYUunAlrXIsJNrcw2BMfpVVQtlBLgjnUIWE75NMKSlfM/Nu2SeRUlJ9VUTV5fcRP+FeOQnSJCcxWmYZQH23faa5Lb2s3ZrhRH28ZCuRziYiHU1u2tmIl17etYlI55+JdDYhvT3u1osF/TMRupr8oM8mPnLe77W5UIMkeNnPBY+5cT/Z08yzw7OWI7G+PyeRnMewStdseib3Ma2Zt7K3z35PMr9/90qBn+pazsznrM+8l8rZLme95vz+Mr9buub7euRtt/ezP0NW8r6sDLU9stdmbeOFbZv1exlmA6lBPAgsFpFDgR8BjwL/DXykkAUbEeqmQrLdtb3jvi9JX4l6Yhd/fVCFlCq+Kr4PqkoKRX1I+YpC8J472atCMjWAk/9AiIBEUS+KUpGXn2fIVPESe4MA4l5eRxPRziZiiWaiiSYEH0EQT9xUsudds5cnbn3m5+txl1nOHWkIEJwkUwnX9Jee+glIBVM/6Zr7OpK9b5OelsJz3fs8EfdzgnY7CCZZf7zdTs7ZwU5C3s/9THBx4ftZwSgscGUFvtxApqmiDmodSIDwVTUpIucC31HV74nIK4Uu2EiQTPm0Vkyls7GZ9rZW2hOpdEZ8ohGPWMQjFhFiEY9oRCiLeMQ8b9SluU+klPZEikTKdyf+9EneJzjBK76C77ugMAp6bvJDBL+sGr+smgT7Ns5GgIgnRDwXLCIieB5EpGtd1BPEgwiCF2wb9fJ0JeOnguCRzryaW8shZ76PK+8e2wTrup3gc662M8+7GD2SvtKRTJFKuR8vElwMRLz0dJA7vGHRoMswkACREJGLgIuBjwXr9ssb3NsTKVo7U+ztSNKWSNGRcFdN4k2hPLk+s50CiZRPIhV+VRX1hGjEoywixKIuaMSCgDLo//Qi6Ez5tHX6tCXc7yGZslN+sSnuhJL0B/d/IeBONhF30ol6gicQ8TwXcETwIl2BJiK9fEfA8oRtAAAbu0lEQVS9iHuZIUkEwaAz4dOR9GlP9v93lf6/8zwhIuB5XvB/J8F6iIhbN9SL0oEEiEuAy4F/U9UNIjID+PnQDjdypHyltTNJa2cqeCXxe6lFazROZ81BlDVtZCB3Nrk/5BTtIXfKRsTVOGJRNy0Lah+xiJe/q71B6kj6tHWmaEukaE+kBn0SMqVLcbW8VHLg/6fZtZX0FW00q/bi9qvdKgeqYeu61ya7PTcns65rGwFEXAAT6VoWIWhmwzW90bWM4E6QQRNbZjsJtkOGvUafSCntyRSdSZ+OhE9Hcmh/U5n/u5S6PEjkv5lvIAPlXge+lLW8AfhG3ktSZO2JFG2dKfZ2JmnrTNGeGNwv2y+rIVk5mWjre/tUDvfHmqI95Kmn6T/MqCd4nruSc8ueu8KLuCuGSESIytD7QdoTfiYYtHWm9r3d34wqQ62tlDJPuq7GPdzfTsTrCj5eJjB1zXteOsBIJvBEcv7usoNBe8KnIzGwv6ekr2xp9tnY6NPUqdSUCdUxcdMyoaYMqmIuKBfSQO5i+ijwNeDgYHsBVFVrC1qyAkr5SlsiRWtHVw0hlYcve7JyAioeoikguFce6NZRiKA97ujIur8+rPMr2I+X2Euko4FkooWBXC144vpC0k0HmeASNClEg2nKVxcUOlO0JZKMor/7QUmklLYktCWVtqTSmlDak9CaVNoS7r3WpNKeUFqTBOvcfHuwfVswH/Ggpkyoirk/6OrMFDfNXhfMV5UJlVG6OpzNsPHV3RiRjy9/ugajaL+781V5v0XZ2JhiY6Ofmb7b7JMcwDVqVYxM0EgHkK4g0jWtKRvad2ogTUzfAc4D/jwiBrLl8INg0BZcDWf3HRRCqmJc4fYdKScVHwt+gkhHI9GOBiTZ+6AyX6FzIN+yUcBXpS0BLQmlpTN4JZTmTmVvZl3P99Mn/rakMtCvhQAVMaiICpVRycxPqPSoiLr5hE/muNv2+qwPjtfazwB8T9yVYTqQVMWEypi4Y8WgMuaOWRnrezkedSepkSrlK9tb3e/u/b0+O1rdDQ5RT4h5EPWCvjyPYFmCdQxom9zlfP6u0k0/3dapsrNNM0FgQ4PPpqYU7zT6tGc9UXVSpTB9TIRjDogyvc5jel2EMXFhb6f7LrekpwmluUNpTpD5Pjd1Kpua/Mw2+TjNDSRAvAusGQnBQVW7BYK2zhQdSX/0PfrAi5GqGE+qYjyS6iDS0UikowFJtRe7ZPsskVIaO5WmDqWhXWnscF/8luBE39ypmT+WzIk/oexN9H/xVxWj21X7AdVecPINTvY58/EoQQDoOkFXRIXyyNCv8lO+K2t22bMD1t5OpTmh7O2E5k5XQ3l/r5+pnexN6ICuLNNBrCt4uNpJer6mTBhTLoyJC3XBdEy5x5i4+1kLLeUru9rcz+aCgJt/v8Ut72jr/+o7n6IelHm4uw49KItAzBNiESgLpm69ez8WkWD73rfrTCnvNPlsaPTZ1Jhib9bFwdi4ML3O4yMfKHOBYIzHwbURKmPhv/vxg7wjW1XpSNEVUDqVT/77EH4vA9jmH4DHROR/yErzrarfGvzh8kdVM23lLhgkaU+MwmDQD42Uk6ycSLJyIpJsJ9LRQKSjEfHzlZF9H8qmrvmlsV1p7PTdtKO3l09DR99X2OURujXHjK3wOKhWqCqDmmBddgDIbuKpjFLw9tqBiHhCbTnUlg+9LJ2pdBOYCxhtCaU16QJP17yrUbUmlL1BE1lrAna3+5ng1BbSzwUQj9AVNOJe90BS7tbVx918XbkQi/T8WXxVdrdrcMIPTv7pYNDis71Vyb5JR4BxFcLkKo85E6JMrnbzk6o8Jld5TKh0d+qkFBI+JP10P4ibT2TNJ33tc5uED6lgOZHq2r4z1bWuM5i69e79lk7NWq8524dfoNSUwfS6CKcdHGN6XSSoFXjUlhe2Z1zEXeDEo8L4yqHvZyAB4t+AFiAOlA39UPmRSPms297ixhzsZ8GgPxqNk4xOJlk1GUm2ZQWLzoIe970Wn6c3JninKdXtpN/U0Xs1N+bBmHKhNjjJTK6KUBecjGqDE1H2tComlIWciPZHZRH3u6jbx6fhtifd/1NDu9LQ4QdTzZr67G7zeXuPW+7t/7IqRiaQlEfINA3lbj82Lkyq8jh8XISTD/KYUu0CwKQqYWKlN6D/36i4q32ntL4Pqayg0elDRNx3fCC1zeyxU2WR4Db44E5GTwRfXeaA9ODS9HwqWJ8epKvBcnrA6b4aSIAYq6pn5uFYeZFMKW2dqf433M9ptIJktIJk1ZRM53akoxG0l8vGQepMKX/YkuTx9Z2s3JZCgCnV7iQxqcpjVr1QF1x1dr28zHyFdcYWXTwqxKPCpCqAvscwqLoaSENHVjBJB5IgmDS0uxrjIWM8PjQ1yuQqj8nVLgBMqvQoH4amq2KKeEKFBxV9BC43YDaSGUAbiwoxz93O3vefg/T3X9SD70MKN3DVH2J73UACxNMicqaqPjWkI+SZVRoGz49V4ceqSFQdgJdoccGis8kN4x+kDQ0pnng7wW82JmjuVCZVChfPKefMGTEmVo2AUX5mSERcU15VmTC1Znj+nwWIxyJUlEUoj3lEXGYs0tfGXWMl0jNdn82Mqcgs97yiVrpvFDYmo9t2ucfotr32WO9J9ywK/QeB/Erfhss+NK1Kf33PItIMVOH6HxIU+TbX+JSZeua/3MmcA+o48oBaDptUQzxmIzgHTdUlj+tsxEt2oFnpCjTIaaNBGuvWJPzPhhaefKuRt3a0EfWED06v48wjJjB/Wi2eF8nc3iupIAme39ljvhADeczo4QlUxKJUlEWIxzzi0choy55RVDJ10QpVXTyYz/RZgxDXBjBbVd/Zp5LlUW1FjJaOJHf/6R0Ud6vaoROrmX1ALbMPqOOIKbVUlxcsi/noIYJfXodfXhf6tqry1rZmnnp9G79bu4P2hM+BYyv5/AkzOPWwidRVdGVb6bqAi6KRPhrG/VSvwSOdXXU01BHTg65Ij+yVrtZySf+bNRI4vb5bvr0s6dvz/ay8VhokO8wedTzSRCNCRSxCPBrUEqLFrIFKLxlWs54REvqK9PG+9Mw2q5qznJuRNiRDbRETIQ6kBrFCVY8apvL0a/a8hXrPY8/R0pHkzfeaeG1rE69tbWTt9haSvhtmNmN8FUceUJupZdRXFr1vfcRobEvw3Fvbeer1bbyzu5V4zOPEmRM484hJHDa5prD9BqpdabhTQfBIBw71ET+J+KmgH2V4T4sCweBC12TQY9BhMKI9/5l8JfxElVnvBlf6WS+l+zpVzwUYBB8vK71GkkQyiZ9Kkkwm8VPuhCTBSUmCk5RkTlL7dqIqi3rEox6VZVHisUhw91NfJ2av+3vdMtNKzjr6eC9s6uVsX+JVlUxK8r6mfp/bSNW4/NYgAn8UkaNV9eXB/kwFEfw/VpdHWTx9LIunjwVcqoy125pZEwSM37y+jV+tdmkvpo6pyNQwZh9Qy8SacusgzeKrsnpzI0+9/j4vrt9F0lcOm1TD1aceyokzx1NZNkw1MhE0UoZGyvpPBxkEjK6pCx7uZJZywSR4PoBoEvH90MASEQlO+C51STRIUpceeZ5JZZKPi9v08xW8KJnnMXR7RXouD/AOmAiD7sPsQdWlz0imlGTwnI2E75MK1iWSKVJBMEmlUu5ZFpoKfrddgSV9Ui8vi1JZHqOivIyq8hjRSJSewcD+DgekSEFsIDWI14FZwCZgL119EPMKX7yejjpqsd7z2LP95kpKpnzW79jLa1sbWbO1kdffa2Jvh+uUHV9dzvRxldTGY1THo9TEo1SXu1dNPNZtuao8WhL3z+eDr0pDa4Ltze3saO5gR3MH25o7WL5xN9ubO6gpj3Lq4RP58BGTmD6+atjLJ0ImMWHMc3d4ZF+VS7ppJrvJJki6lknUJt33l93EIyJBwAiCRuaKK1vI30Po30g/26WfQZD90J1RpLdggkJleZTKWARvlPzdjBZBa9CgahADCRAHh61X1U2DOVC+LF68WF9++WW2NbkT3ED5qmza1RoEjCbea2ijpSNJc7tLWd2XqvIINeUumLggkhVMgkCSHWgqy1xHW0UsMqzBJZHyMyf+Hc0dbG9uZ3tzBztautblJlirKo8wc2INZx45iWNnjKOsQO3AnkcmpXnUE8qibtqV7tzd+22MKYyCBIhSs3jxYl2+fDkAezuSbN7Tts/5hpIpn5aOpHu1J2kOAkdLRyKznD11gSVBS0f/ie3iMY/K4M6MyswrezmaWV8RC6ZZ69LvRz1hb2eKHemTfnNHZpoOBntyhiELUF9VxsSacibWlDOhppwJNXE3X13OxNryvDUfRTyhqjwSDPaRoAYQBIOIZ1eTxhTZUALEiK73VpVHmTmxmvea2tndMvTRwtGIx5jKMsYMsjPbVzdoryUTUFzgaAseLNQWPEcinTE2vb6xsa37cyYGEKM96TmUPxaR4EQfZ/H0sW4+EwzijKsuI1bgq3LPgwnV5YyvLrcgYMwoM6IDBIDnCVPHVFAbj7J5T9uwPt3ME6Eq6KeYNMRRIaqaeSBPOmC0ZgJMMhNo2hIpauJRJtbEg5pAOWMqYkXrbBeBsUHtxJqGjBmdRnyASKuJx5g1KcrWhjYa+supXEJEhHgsQjwWoX74+4WHZExljEm18YL1VxhjSsOoCRDg2sEPHFtJbTzBloa2vDwEyHSpiUeZXBe3kevG7CdGVYBIq6uMUVkeYcueNprDnt1pBqWiLMKUujhVNkLdmP3KqP2Lj0U8po+vYvfeTrY2tFlq8CEoj3lMqo13S6thjNl/jNoAkTa2qoyq8gib97TR2mFpwgciFhUm1sSpryxeJ7gxpvgK1ssoIneIyHYRWdPL+yIi3xWRdSKyWkQWFaos5dEIH5hQzeS6uI3s74PnweS6OLMm1jC2qsyCgzH7uULehvJj4Kw+3j8bmBm8LgVuLWBZAJhQU86hE6upKLO7b7KJuN/N4ZNrmVBj4xmMMU7BmphU9XkRmd7HJucAP1U3lPuPIjJGRKao6nuFKhO4B5B8YEI1Da0JGtrcSOn9VTzmUVsRY2xV4QfUGWNGnmL2QUwF3s1a3hys6xEgRORSXC2Dgw46aJ8PLCLUV5VRX1VGIuW7YNHa2W8CwNGgosyjNh6jtiJmt6saY/pUzAAR1o4Req+Rqi4DloHLxZTPQsQiXmZkcnsixZ7WThpaE8M6IrvQKsoi1FZEqauIUR61oGCMGZhiBojNwIFZy9OArUUqC+Can6bUVTClroLm9gQNrQka2xIj8hbZyvIItfEYdRUxG/FsjBmSYgaIR4GrReQe4FigsdD9D4PhngsRY6qvNLUn2NNa+v0VleUR6ipcULA+BWPMvipYgBCRu4FTgPEishm4geA5Yap6G/AY8BFgHdAKXFKosuwLz5NMptdS668QgcoyFxRqLSgYY/KskHcxXdTP+wpcVajjF0Ip9FeIuDTndRUxauNRy6RqjCmYUT+SulDy0V8h0vVYTM8LpplnqAte8DhNT9xjM6vLo9RWxEbNI1CNMaXNAkQeZPdX7O10/RTuBO9O7tB1ks+c9O0kb4wpcRYg8sjzhJq4JbYzxowO1oBtjDEmlAUIY4wxoSxAGGOMCWUBwhhjTCgLEMYYY0JZgDDGGBPKAoQxxphQFiCMMcaEsgBhjDEmlAUIY4wxoSxAGGOMCWUBwhhjTCgLEMYYY0JZgDDGGBPKAoQxxphQFiCMMcaEsgBhjDEmlAUIY4wxoSxAGGOMCWUBwhhjTCgLEMYYY0JZgDDGGBPKAoQxxphQFiCMMcaEKmiAEJGzROQtEVknIteFvL9URHaIyKrg9YVClscYY8zARQu1YxGJALcAHwY2Ay+LyKOq+nrOpveq6tWFKocxxpihKWQN4hhgnaq+raqdwD3AOQU8njHGmDwqZICYCrybtbw5WJfrkyKyWkQeEJEDC1geY4wxg1DIACEh6zRn+ZfAdFWdBzwN/CR0RyKXishyEVm+Y8eOPBfTGGNMmEIGiM1Ado1gGrA1ewNV3aWqHcHi7cBRYTtS1WWqulhVF0+YMKEghTXGGNNdIQPEy8BMEZkhImXAEuDR7A1EZErW4seBNwpYHmOMMYNQsLuYVDUpIlcDTwIR4A5VfU1EbgKWq+qjwJdE5ONAEtgNLC1UeYwxxgyOqOZ2C5S2xYsX6/Lly4tdDGOMGVFEZIWqLh7MZ2wktTHGmFAWIIwxxoSyAGGMMSaUBQhjjDGhLEAYY4wJZQHCGGNMKAsQxhhjQlmAMMYYE8oChDHGmFAWIIwxxoSyAGGMMSaUBQhjjDGhLEAYY4wJZQHCGGNMKAsQxhhjQlmAMMYYE8oChDHGmFAWIIwxxoSyAGGMMSaUBQhjjDGhLEAYY4wJZQHCGGNMKAsQxhhjQlmAMMYYE8oChDHGmFAWIIwxxoSyAGGMMSaUBQhjjDGhChogROQsEXlLRNaJyHUh75eLyL3B+y+JyPRClscYY8zAFSxAiEgEuAU4GzgSuEhEjszZ7PPAHlU9FPg28O+FKo8xxpjBKWQN4hhgnaq+raqdwD3AOTnbnAP8JJh/ADhdRKSAZTLGGDNA0QLueyrwbtbyZuDY3rZR1aSINALjgJ3ZG4nIpcClwWKLiLxVkBLD+Nxjl7iRVl4YeWUeaeUFK/NwGGnlBThssB8oZIAIqwnoELZBVZcBy/JRqL6IyHJVXVzo4+TLSCsvjLwyj7TygpV5OIy08oIr82A/U8gmps3AgVnL04CtvW0jIlGgDthdwDIZY4wZoEIGiJeBmSIyQ0TKgCXAoznbPApcHMyfD/xWVXvUIIwxxgy/gjUxBX0KVwNPAhHgDlV9TURuApar6qPAj4Cficg6XM1hSaHKM0AFb8bKs5FWXhh5ZR5p5QUr83AYaeWFIZRZ7ILdGGNMGBtJbYwxJpQFCGOMMaH2ywAhIneIyHYRWZO1bqyI/EZE1gbT+mKWMZeIHCgiz4rIGyLymoh8OVhfkuUWkbiI/ElEXg3Ke2OwfkaQVmVtkGalrNhlzSUiERF5RUR+FSyXdJlFZKOI/FlEVqVvZSzV7wWAiIwRkQdE5M3g+/zBEi/vYcHvNv1qEpFrSrzM/yv4u1sjIncHf4+D/h7vlwEC+DFwVs6664BnVHUm8EywXEqSwN+r6hHAccBVQeqSUi13B3Caqs4HFgBnichxuHQq3w7KuweXbqXUfBl4I2t5JJT5VFVdkHVvfql+LwD+H/CEqh4OzMf9rku2vKr6VvC7XQAcBbQCD1OiZRaRqcCXgMWqOgd3k9AShvI9VtX98gVMB9ZkLb8FTAnmpwBvFbuM/ZT/F8CHR0K5gUpgJW4k/U4gGqz/IPBkscuXU9ZpuD/204Bf4QZzlnqZNwLjc9aV5PcCqAU2ENwgU+rlDSn/mcDvS7nMdGWoGIu7U/VXwF8N5Xu8v9YgwkxS1fcAgunEIpenV0HW24XAS5RwuYOmmlXAduA3wHqgQVWTwSabcV/mUvId4B8AP1geR+mXWYGnRGRFkJYGSvd7cQiwA7gzaMb7oYhUUbrlzbUEuDuYL8kyq+oW4JvAO8B7QCOwgiF8jy1AjDAiUg08CFyjqk3FLk9fVDWlrlo+DZe88YiwzYa3VL0TkY8C21V1RfbqkE1LpsyB41V1ES5z8lUiclKxC9SHKLAIuFVVFwJ7KZGmmf4EbfYfB+4vdln6EvSFnAPMAA4AqnDfjVz9fo8tQHTZJiJTAILp9iKXpwcRieGCw12q+lCwuuTLraoNwHO4vpMxQVoVCE+/UkzHAx8XkY247MOn4WoUpVxmVHVrMN2Oaxs/htL9XmwGNqvqS8HyA7iAUarlzXY2sFJVtwXLpVrmM4ANqrpDVRPAQ8CHGML32AJEl+y0Hxfj2vhLhogIbuT5G6r6ray3SrLcIjJBRMYE8xW4L+0bwLO4tCpQQuUFUNV/UtVpqjod15TwW1X9DCVcZhGpEpGa9DyujXwNJfq9UNX3gXdFJJ1Z9HTgdUq0vDkuoqt5CUq3zO8Ax4lIZXDeSP+OB/89LnaHSpE6ce7Gtc0lcFc0n8e1NT8DrA2mY4tdzpwyn4CrEq4GVgWvj5RquYF5wCtBedcA1wfrDwH+BKzDVdXLi13WXsp/CvCrUi9zULZXg9drwL8E60vyexGUbQGwPPhuPALUl3J5gzJXAruAuqx1JVtm4EbgzeBv72dA+VC+x5ZqwxhjTChrYjLGGBPKAoQxxphQFiCMMcaEsgBhjDEmlAUIY4wxoSxAmBFBRMZlZdN8X0S2ZC0PKLuqiNyZdf99b9tcJSKfyVOZzwnK96qIvC4iX+hn+9OChIZh700Rkcey9vVosP5AEbk3H+U1Jpfd5mpGHBH5KtCiqt/MWS+477Qf+sFhJCLluKR0i1V1a7B8sKr+pY/P3AzsVNXvhLz3I9wo3luC5XmqurpAxTcGsBqEGeFE5NAg5/1tuIyxU0RkmYgsD/LhX5+17QsiskBEoiLSICLfCK7IXxSRicE2N4vINVnbf0Pccy3eEpEPBeurROTB4LN3B8dakFO0Olwep90AqtqRDg4iMklEHgo+9ycROU5EPgB8Abg2qHV8KGd/U3CDOgn2tzrr518VzN+ZVavaKSL/Eqy/LjjO6uzfhzH9sQBhRoMjgR+p6kJ1mSyvU/dchPnAh4PnZuSqA/5H3fMqXgT+tpd9i6oeA1wLpE+ufwe8H3z2G7jMut2oy4v0JLBJRP5bRC4SkfTf23eB/wjK+Cngh6q6Hvgh8H/VPXvgDzm7/D7wExH5rYj8czoHUM4xL1GXHPFcXGrnn4rIR4CDcKnWFwAfCgk+xoSyAGFGg/Wq+nLW8kUishJXozgCF0Bytanq48H8CtzzQcI8FLLNCbhkfqhqOsVFD6q6FPfMjuW4jKXLgrfOAG4LrvwfAeqDfFW9UtXHgA/g8nEdCbwiIuNytwv2cz9whaq+i8vNdDYu7clK4FBgVl/HMiYt2v8mxpS8vekZEZmJeyLcMaraICI/B+Ihn+nMmk/R+99CR8g2YSnAQwVNQatF5L9xyQq/EHz+GFXNLgOuC6XPfe0C7gLuEpEncIEqNzjdDtyjqs9mlfVmVf3RQMtsTJrVIMxoUws0A01BM8xfFeAYL+CahhCRuYTUUESkVro/l2EBsCmYfxq4KmvbdP9FM1ATdkAROT1dyxCRWlyu/3dytvkyEMvpvH8S+HyQ6RURmSYi4wf4c5r9nNUgzGizEpfaeA3wNvD7Ahzje7j2/dXB8dbgntqVTYB/EpHbgTagha5+jquAW0XkEtzf4LPBul8A94vIecBVOf0QRwPfF5EE7sLuVlV9RUQOzdrmK0BrutMa+L6q/lBEDgf+GNRQmoFP4/oojOmT3eZqzCCJe+hKVFXbgyatp4CZ2vU4R2NGBatBGDN41cAzQaAQ4DILDmY0shqEMcaYUNZJbYwxJpQFCGOMMaEsQBhjjAllAcIYY0woCxDGGGNC/f/x1kWgzUS9WAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f457f6654e0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_learning_curve(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows a typical learning curve: for very few training points, there is a large separation between the training and test error, which indicates **over-fitting**.  Given the same model, for a large number of training points, the training and testing errors converge, which indicates potential **under-fitting**.\n",
    "\n",
    "As you add more data points, the training error will never increase, and the testing error will never decrease (why do you think this is?)\n",
    "\n",
    "It is easy to see that, in this plot, if you'd like to reduce the MSE down to the nominal value of 1.0 (which is the magnitude of the scatter we put in when constructing the data), then adding more samples will *never* get you there.  For $d=1$, the two curves have converged and cannot move lower. What about for a larger value of $d$?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJztvXmcXFWZ//9+aumq3qvS2fcAYUvIRlgUZZcBZ4QBUYP6HYKjLOo4OCMz6Hx/IBG/wziOMoLCKyjMODLIIiAqi4IgosgSCDEQIAkkZN87vS9V9fz+OLeqq6uruqu7q7qrK8/79arce889996nK1XnU885z3mOqCqGYRiGkYlvtA0wDMMwShMTCMMwDCMrJhCGYRhGVkwgDMMwjKyYQBiGYRhZMYEwDMMwslI0gRCRsIi8KCKvicjrInJDljohEblXRDaIyAsiMrtY9hiGYRiDo5geRCdwpqouBBYB54rIyRl1/hY4oKpHAN8F/q2I9hiGYRiDoGgCoY4W7zDovTJn5V0A/Le3/wBwlohIsWwyDMMw8idQzJuLiB9YBRwBfF9VX8ioMg3YAqCqMRE5CDQAezPuczlwOUB1VeXxRx8xu3BGKrDvbaiMQvWE3PUCIRB/4Z5rGIYxgqxatWqvqvbTyPWlqAKhqnFgkYhEgIdEZL6qrk2rks1b6JP7Q1VXAisBli48Vl9+7O7CGvrAZ6BmIpx7U+469TOgenxhn2sYhjFCiMjmwV4zIlFMqtoIPAOcm3FqKzADQEQCQD2wfyRs6kVkFhx4r/868e6RscUwDKNEKGYU0wTPc0BEKoGzgTczqj0CXOrtXwz8Vkcje2B0FjTvgFhn7joJEwjDMA4tiulBTAGeFpE1wEvAb1T1lyKyQkTO9+r8CGgQkQ3APwDXFtGe3ERmAQqN/XgR5kEYhnGIUbQxCFVdAyzOUn5d2n4H8LFi2ZA30Vlu2/gejJ+bvY4JhFEmdHd3s3XrVjo6OkbbFKMIhMNhpk+fTjAYHPa9ijpIPWaonw7igwObcteJd42YOYZRTLZu3UptbS2zZ8/GosrLC1Vl3759bN26lTlz5gz7fpZqA8BfAXVTobGfQX6NQyIxcjYZRpHo6OigoaHBxKEMEREaGhoK5h2aQCSJzOp/DAJsoNooG0wcypdC/t+aQCSJzoLGLZCI5a5j3UyGYRxCmEAkicxy3UhN23LXsYFqwxg2jY2N/OAHPxjStR/+8IdpbGzst851113Hk08+OaT7G70xgUiSjGTqb8Jcf96FYRh50Z9AxOPxfq999NFHiUQi/dZZsWIFZ5999pDtKxYD/W2liAlEkshMt7VIJsMoKtdeey0bN25k0aJFXHPNNTzzzDOcccYZfPKTn+S4444D4K//+q85/vjjmTdvHitXrkxdO3v2bPbu3cumTZs45phj+NznPse8efM455xzaG9vB2D58uU88MADqfrXX389S5Ys4bjjjuPNN91c3T179vChD32IJUuWcMUVVzBr1iz27u2VAo54PM7y5cuZP38+xx13HN/97ncB2LBhA2effTYLFy5kyZIlbNy4EVXlmmuuSdW99957AbL+bT/5yU848cQTWbRoEVdccQXxeDzns0YbC3NNEqyC6on9RzJZF5NRZtzwi9d5Y3tTQe957NQ6rv/IvJznb7rpJtauXcvq1asB14i++OKLrF27NhWaeeeddzJu3Dja29s54YQT+OhHP0pDQ0Ov+6xfv5577rmHO+64g49//OP87Gc/49Of/nSf540fP55XXnmFH/zgB3z729/mhz/8ITfccANnnnkmX/3qV3n88cd7iVCS1atXs23bNtaudenjkl1bn/rUp7j22mu58MIL6ejoIJFI8OCDD7J69Wpee+019u7dywknnMCpp54K0OtvW7duHffeey9/+MMfCAaDfP7zn+fuu+9m3rx5WZ812pgHkU50gEgmEwjDKAonnnhir7j9733veyxcuJCTTz6ZLVu2sH79+j7XzJkzh0WLFgFw/PHHs2nTpqz3vuiii/rUee6551i2bBkA5557LtFotM91hx12GO+88w5/93d/x+OPP05dXR3Nzc1s27aNCy+8EHCT0qqqqnjuuee45JJL8Pv9TJo0idNOO42XXnqpz9/21FNPsWrVKk444QQWLVrEU089xTvvvJP1WaWAeRDpRGbBul+AJtzEuUwszNUoM/r7pT+SVFdXp/afeeYZnnzySZ5//nmqqqo4/fTTs8b1h0Kh1L7f7091MeWq5/f7icXcOGI+Kd+i0SivvfYaTzzxBN///ve57777uPnmm7PW7e9+6X+bqnLppZfyr//6r33qZT7rzjvvHNDGYmMeRDrRWRDvhJZd2c+bB2EYw6a2tpbm5uac5w8ePEg0GqWqqoo333yTP/3pTwW34QMf+AD33XcfAL/+9a85cOBAnzp79+4lkUjw0Y9+lG984xu88sor1NXVMX36dB5++GEAOjs7aWtr49RTT+Xee+8lHo+zZ88enn32WU488cQ+9zzrrLN44IEH2L17NwD79+9n8+bNWZ9VCpgHkU56JFPtlCwV1ImEf/g5TgzjUKWhoYFTTjmF+fPnc9555/GXf/mXvc6fe+653H777SxYsICjjjqKk0/OXKl4+Fx//fVccskl3HvvvZx22mlMmTKF2traXnW2bdvGZZddRsLLoJD81f8///M/XHHFFVx33XUEg0Huv/9+LrzwQp5//nkWLlyIiPCtb32LyZMnpwbFkxx77LHceOONnHPOOSQSCYLBIN///veprKzM+qzRRkYju/ZwKMqCQUk6DsKPL4CTr4IFn8heZ/xRUFFVnOcbxgiwbt06jjnmmNE2Y1Tp7OzE7/cTCAR4/vnnueqqq1KD5uVAtv9jEVmlqksHcx/zINIJ10M4Agf6iWSycQjDGPO89957fPzjHyeRSFBRUcEdd9wx2iaVJCYQmVgkk2GUPXPnzuXVV18dbTNKHhukziQyy02Wy9X1ZgJhGMYhgglEJtFZ0NUC7TmWxrbZ1IZhHCKYQGQSSVtdLhuWj8kwjEMEE4hMUqGum7Kfty4mwzAOEUwgMqka7/Iy5Ypksi4mwxhxampqANi+fTsXX3xx1jqnn346L7/8cr/3ufnmm2lra0sd55M+/FDGBCITkf4jmWzpUcMYNaZOnZrK1DoUMgUin/Tho0GppAY3gchGMpIpFzYXwjCGzD//8z/3Wg/i61//Ov/xH/9BS0sLZ511Vio1989//vM+127atIn58+cD0N7ezrJly1iwYAGf+MQneuViuuqqq1i6dCnz5s3j+uuvB1wCwO3bt3PGGWdwxhlnAD3pwwG+853vMH/+fObPn5/KudRfWvF07r//fubPn8/ChQtTWVzj8Thf+cpXOO6441iwYAG33HIL4BL2LV68mOOOO47PfOYzdHZ2pmxZsWIFH/jAB7j//vvZuHEj5557Lscffzwf/OAHU7Oysz2rWNg8iGxEZsHbj0NnM4Rq+56Pd0Eg1LfcMMYaj10LO/9c2HtOPg7Ouynn6WXLlnH11Vfz+c9/HoD77ruPxx9/nHA4zEMPPURdXR179+7l5JNP5vzzz8+5xvJtt91GVVUVa9asYc2aNSxZsiR17pvf/Cbjxo0jHo9z1llnsWbNGr70pS/xne98h6effprx48f3uteqVau46667eOGFF1BVTjrpJE477TSi0WheacVXrFjBE088wbRp01JdVitXruTdd9/l1VdfJRAIsH//fjo6Oli+fDlPPfUURx55JH/zN3/DbbfdxtVXXw247LDPPfcc4PI23X777cydO5cXXniBz3/+8/z2t7/N+qxiYR5ENqIDRDLZQLVhDJnFixeze/dutm/fzmuvvUY0GmXmzJmoKl/72tdYsGABZ599Ntu2bWPXrhyJM4Fnn3021VAvWLCABQsWpM7dd999LFmyhMWLF/P666/zxhtv9GvTc889x4UXXkh1dTU1NTVcdNFF/P73vwfySyt+yimnsHz5cu64445U99CTTz7JlVdeSSDgfoePGzeOt956izlz5nDkkUcCcOmll/Lss8+m7vOJT7gUPy0tLfzxj3/kYx/7WGphoR07duR8VrEwDyIb6ZFMk7KkQzaBMMqFfn7pF5OLL76YBx54gJ07d6bWZbj77rvZs2cPq1atIhgMMnv27KxpvtPJ5l28++67fPvb3+all14iGo2yfPnyAe/TX066fNKK33777bzwwgv86le/YtGiRaxevRpV7WPfQLnvkqnBE4kEkUgka36obM/KXEypUJgHkY2ayS5ja67V5WwMwjCGxbJly/jpT3/KAw88kIpKOnjwIBMnTiQYDPL000+zeXM/OdGAU089lbvvdok7165dy5o1awBoamqiurqa+vp6du3axWOPPZa6Jleq8VNPPZWHH36YtrY2Wltbeeihh/jgBz+Y99+zceNGTjrpJFasWMH48ePZsmUL55xzDrfffntqDYr9+/dz9NFHs2nTJjZs2AC4zLCnnXZan/vV1dUxZ84c7r//fsAJy2uvvZbzWcXCPIhs+PxujeoD1sVkGMVg3rx5NDc3M23aNKZMcan1P/WpT/GRj3yEpUuXsmjRIo4++uh+73HVVVdx2WWXsWDBAhYtWpRaf2HhwoUsXryYefPmcdhhh3HKKaekrrn88ss577zzmDJlCk8//XSqfMmSJSxfvjx1j89+9rMsXrw45yp1mVxzzTWsX78eVeWss85i4cKFzJ8/n7fffpsFCxYQDAb53Oc+xxe/+EXuuusuPvaxjxGLxTjhhBO48sors97z7rvv5qqrruLGG2+ku7ubZcuWsXDhwqzPKhZFS/ctIjOAHwOTgQSwUlX/M6PO6cDPgXe9ogdVdUV/9y1quu90nloBu9+AS37a91ywGiYcWXwbDKMIWLrv8mcspPuOAf+oqq+ISC2wSkR+o6qZo0W/V9W/KqIdQyM6GzY+DbEOCIR7n7MuJsMwDgGKNgahqjtU9RVvvxlYB0wr1vMKTmQmoNCYpX/PupgMwzgEGJFBahGZDSwGXshy+n0i8pqIPCYipbGCOvQk7cs6YU5NJIwxzVhbSdLIn0L+3xZdIESkBvgZcLWqNmWcfgWYpaoLgVuAh3Pc43IReVlEXt6zr+/i4kWhfjqIL3ckkwmEMUYJh8Ps27fPRKIMUVX27dtHOBweuHIeFDWKSUSCOHG4W1UfzDyfLhiq+qiI/EBExqvq3ox6K4GV4Aapi2lzCn8Q6qb1E8nUBdja1MbYY/r06WzdupU9e/aMtilGEQiHw0yfPr0g9yqaQIibIfIjYJ2qfidHncnALlVVETkR59HsK5ZNgyY6Cxo3ZT9n60IYY5RgMMicOXNG2wxjDFBMD+IU4P8AfxaR5HTArwEzAVT1duBi4CoRiQHtwDItJb83Mhs2P+/EwJfxVlkXk2EYZU7RBEJVnwOyZ9nqqXMrcGuxbBg20ZkuvffBbT3pN5LYuhCGYZQ5lmqjP/qLZLIuJsMwyhwTiP6IzHTbbJFM5kEYhlHmmED0R7ASaiZlT/ttYxCGYZQ5JhADEc2xupwtPWoYRpljAjEQkdku3YZmEQPrZjIMo4wxgRiI6EyId0JzlpWtLGmfYRhljAnEQCQjmbJNmLNxCMMwyhgTiIFIhbpmi2QygTAMo3wxgRiIcB1URrNHMlkXk2EYZYwJRD5EckQy2SC1YRhljAlEPkRnu8lymWmi4jab2jCM8sUEIh8iM6GrFdr39y63LibDMMoYE4h8iObIyWSD1IZhlDEmEPkQne22fSKZbOlRwzDKFxOIfKgcBxXVlpPJMIxDChOIfBCxSCbDMA45TCDyJRnJlImtC2EYRpliApEvkZnQfgA6mnqXWxeTYRhliglEviQjmTK9COtiMgyjTDGByJfIbLfNFAjrYjIMo0wxgciX2kngD8GBjEgm8yAMwyhTTCDyRXwQmdE37beNQRiGUaaYQAyG6Oy+k+Vs6VHDMMoUE4jBEJkJLbugu713uXUzGYZRhphADIZkyo3MGdWWtM8wjDLEBGIwRHKFuppAGIZRfphADIb6aSD+LJFMJhCGYZQfJhCDwReA+ul9I5msi8kwjDKkaAIhIjNE5GkRWScir4vI32epIyLyPRHZICJrRGRJsezphS/gPIGhEJnVN5LJBqkNwyhDiulBxIB/VNVjgJOBL4jIsRl1zgPmeq/LgduKaI9D/DDucKiMDO366Exo2ta7W8mWHjUMowwpmkCo6g5VfcXbbwbWAdMyql0A/FgdfwIiIjKlWDYhPmg4HCqqIDxUgZgNmoCDW3vKrIvJMIwyZETGIERkNrAYeCHj1DRgS9rxVvqKCCJyuYi8LCIv79l3YIhG+GDcYW7hH4BQLfiCg79PtkgmG6Q2DKMMKbpAiEgN8DPgalVtyjyd5RLtU6C6UlWXqurSCQ3RoVjhfvmHatMNg3D94G8VmeHu12suhC09ahhG+VFUgRCRIE4c7lbVB7NU2QrMSDueDmwvsBVOHLKJQeUQxCYQhtrJfVeXM4EwDKPMKGYUkwA/Atap6ndyVHsE+Bsvmulk4KCq7iioIZGZuQekQzXgrxjCPWfZuhCGYZQ9gSLe+xTg/wB/FpHVXtnXgJkAqno78CjwYWAD0AZcVlAL6mdC1bj+64Qj0Lp7cPeNzoTtr0AiDj4vXNbWhTAMo8womkCo6nNkH2NIr6PAF4piQN00qG4YuF5ldPACEZntPIaWne45YB6EYRhlR3nOpK6dAjUT86tbUeXGFQZDcvnRAxbJZBhG+VJ+AlEzyQ0iD4bBzomIzHTb9Egm62IyDKPMKC+BqJ4AdVMHf91go5lCtVDV0DuSybqYDMMoM8pHICrHuUR6QyEYhkDl4K7JjGSyLibDMMqM8hCIcKRnXGCoDNaLiMx0ab/Vm9dnS48ahlFmjH2BCNf3rPQ2HAabvC86G7pboW1vT5l1MxmGUUaMbYEI1UF0jkubMVwCIQhW518/WySTJe0zDKOMGLsCUVFTOHFIMphuplQk0wiNQ1hKccMwRph+BUJE/CLy7yNlTN4Eq1xmVl+B9W0w3UyV41w0UzHnQqhC+wHYuwF2v24D4YZhjCj9zqRW1biIHC8i4s16Hn3E5xb88Q1xRbj+8Aehoha6mvOwQ/pGMhWqiyneDW37oHVv73u27Bp6pJZhGMYgySfVxqvAz0XkfqA1WZgjO2vxCYTAX8QUUpWR/AQCXDfT5j/2HA93kLqzBVr3QMdBsmQ9d6JRM8kJmWEYRpHJp6UdB+wDzkwrU2B0BKL/9E7DJxzxVovLw2GKzoa3HnUNerh+aOMEiQS073feQqy9/7qaMC/CMIwRY0CBUNXCZlgtdfwBN7bQmbm2URbSV5ebvGBwHkR3hwuRbdvv5lDki3kRhmGMEAOO8orIdBF5SER2i8guEfmZiJT3T9h8o5miXiRTcqA6EeuZOJcNVWhvdIPOe9a57qTBiAP0eBGGYRhFJp8woLtwC/tMxa0X/QuvrHwJ17vB8IGomeQywaYimTR70r54NzTvhN1vwIF38x/jyEXbPotoMgyj6OQjEBNU9S5VjXmv/wImFNmu0cXn771+dS7E5waqe82FSOtm6myB/e/CrteheUfhZlqbF2EYxgiQj0DsFZFPe3Mi/CLyadygdXmTbzdTZGbvuRCxTjfgvPtN2LceOhrJa8B7sJgXYRhGkclHID4DfBzYCewALvbKyptQPUgecy2is92KdN1t7rhxMxzcMnBE0nAxL8IwjCLTbxSTiPiBj6rq+SNkT+ng80G4zs1k7o9UJNN7MOHo4tuVjkU0GYZRRPr1IFQ1DlwwQraUHvl0M2VL2jdSmBdhGEYRyWei3B9E5FbgXnrPpH6laFaVCqE68AX6X060bqqr0zgKAgHmRRiGUTTyEYj3e9sVaWVK75nV5YmIC3lt62dM3hdwM5tHw4MAm11tGEbRGGgMwgfcpqr3jZA9pUdltH+BADcOsX/jyNiTDfMiDMMoAgONQSSAL46QLaVJqBZ8AzS80VnQtH30VpSzsQjDMIpAPmGuvxGRr4jIDBEZl3wV3bJSYqB1IiKzXCN9cOvI2JMNmxdhGEaByWcMIjnn4QtpZQocVnhzSpRwxOVNykV6JNO4UXpbbCzCMIwCk0821zkjYUhJE6oBf0XuLqT6GYCMXiRTEhuLMAyjgOSTzbVKRP6viKz0jueKyF/lcd2dXgbYtTnOny4iB0Vktfe6bvDmjyD9zYkIhKB2yuhFMiWxsQjDMApIvtlcu+gJd90K3JjHdf8FnDtAnd+r6iLvtWKAuqNLeIBxiOis0fcgwMYiDMMoGPkIxOGq+i2gG0BV28ljWTdVfRbYPzzzSoiKKpfaOxeRWS4HU2KQ6zsUGvMiDMMoEPkIRJeIVOKlJBWRw4HOAj3/fSLymog8JiLzclUSkctF5GUReXnPnn4Gi4tNf15EdJa37sOOkbMnF+ZFGIZRAPIRiOuBx4EZInI38BTwTwV49ivALFVdCNwCPJyroqquVNWlqrp0woRRXIqiv3GI0czJlIl5EYZhFIABBUJVfwNcBCwH7gGWquozw32wqjapaou3/ygQFJHxw71vUQmGIViV/VzEW360FMYhwLwIwzCGTT7zIFDVfcCvCvlgEZkM7FJVFZETcWJV+gsRhSM9az+kU1EDVeOLJxCqLjdU3vVtXoRhGMMjL4EYCiJyD3A6MF5EtuK6qoIAqno7buGhq0QkBrQDy1S1CEuvFZjKKDRvz34uOiv/LiZV6Gpx6020N7ptR2Pu4+52OPkqmH9R/rbavAjDMIZB0QRCVS8Z4PytwK3Fen7RCFRAsBq6W/uei8yCtx9zy412pDX0uQRAc0Q8hepceo/KKETnwNQoHHgXnr8VJhwFk3KO5/dGE9CyG+qnDf3vNQzjkGVAgfCilraqaqeInA4sAH6sqo3FNq5kqYxmF4hxc9wv/Yev7F0eCLtrKiNQPRHGH+n2w9EeIUhuw/UuhXgmXS3ws8/BUyvgojvcanf50LYXaiaaF2EYxqDJx4P4GbBURI4AfgQ8Avwv8OFiGlbSVEagKUtivrkfcoPYwcqeRj8cccfDpaIGzroeHvki/O7f4Jwb8xuTMC/CMIwhkk+Ya0JVY8CFwM2q+mVgSnHNKnH8Qaio7VseCMMRZ8Gs98PEY1z6jUKIQ5KJR8NJV8DmP8DrD+Z/Xdtei2gyDGPQ5CMQ3SJyCXAp8EuvzPor8lmvuhjMvxhmnQJ/ug32vJnfNUkvwjAMYxDkIxCXAe8Dvqmq74rIHOAnxTVrDBCuJ4+MI4VHBE77Z6gcB0+ucGMT+WBehGEYgySfiXJvqOqXVPUe7/hdVb2p+KaVOP6AW21uNAjXwVnXQctOePY/XMjsQJgXYRjGIMkn3fdficirIrJfRJpEpFlEmkbCuJJntLqZACbPhxM+C+88Det+kd815kUYhjEI8uliuhk3/tCgqnWqWquqecZYljnhepB83sIisXAZTD8Bnr8F9m0YuL55EYZhDIJ8WrctwNoxMct5pPH53aS20UJ8cMbXnA1P3pA9BUgm5kUYhpEn+QjEPwGPishXReQfkq9iGzZmqBxgIaGiPz8KZ/5/0LQNfv/dgccjzIswDCNP8hGIbwJtQBioTXsZAKF6EP/o2jB1ESy5FDb8Bt5+fOD65kUYhpEH+cykHqeq5xTdkrGKz+fGItpHefG8xZ+GHavhuZvdJL3o7Nx1bXa1YRh5kI8H8aSImED0x2h3M4EbDznz/7qlUZ/8OsQ6+q9vXoRhGAOQj0B8AXhcRNotzDUHobrsCfZGmqoGOONfXMrxP97Sf10bizAMYwD6FQgREWCeqvpUtdLCXHMg4s2sLgGmL4VFn4I3fwUbnuy/rnkRhmH0Q78C4YW2PjRCtoxtRnPSXCZLl8Ok+fD7/4DGLbnrmRdhGEY/5NPF9CcROaHolox1QrXgK5Echr6AS8XhC8JTN0CsM3fdtr3QtN0JRdt+6GiCrjbnWdjUF8M4pMmn4/wM4AoR2Qy04jLUqaouKKplY5HKCLTuGW0rHDUT4fRr4YmvucyvH7g6e73k2tW58AW8V9ANhPuD/RyP4qxywzAKTj4CcV7RrSgXKqOlIxDg1qVY8HFYc5+bK3HY6YO/RyLmXgwQFQVuPog/CP4KJxr+Cu846ATEX+GSHBqGMSYY8NuqqptHwpCyoKLaNYLxrtG2pIcTPgc7/gy/+3e31Gnd1OI9S+MQi/cfYiu+3mLhr/COgz3iYsujGkZJYD/nCk10DnQ2QWeLW7daE6Nrjz/oxiMe/Kxbz/r8W0a3AdYExDvdKyeS5nV4L8SJS6+XZGyzvJDBdX0l4u6lCSd4ibi39Y41kVaWfpxRX3zOo/IFXFecz9vvtyxQWt10qt77Ect4L9L3Y2nvQ8xdk3zvfd57ID73t6beE7/3f5Z+Ln1/FNZZKVdUez6jQ8AEotBUVLlXLe4/p6vVe7W412gIRt0Ut8jQb66DF1fC+74w8jYMCnVeWLwLChWFmykayX3NEIRCoQnviznYP0DSBCTg2eo1mKmGM1mWXj7IMtXeDX8i1tOQJMtH7ceNZBGUPISz36CKfAMuMt/rzNPp5dnqSkZdyb3f5/+zn/3UZzT9lSzTtM9bRr28/+7smEAUExEI1bgXk9x/ZHebE4zOZrctZKPUH3NOhXkXwp/vhymLYPYpI/PcUiH1hSl11InKoIWlnPDEyxh1TCBGEhE3TlFR7aKMwIWUdrVClycYxfxinHQl7FwLv7sJxv8QaiYV71mGYYx5SqjD8xClogpqJsC4w2DycTDhaKifAeFI4edVBEJw9vVOhJ76hv1KMwyjX0wgSo1gJVSPh3Fz3LKiE46B+plOMApB/XT44Fdg11p4+c7C3NMwjLLEBKLUCYahusEJRsMRLgx0uBxxFhz9V7D6f2HLi8O/n2EYZUnRBEJE7hSR3SKyNsd5EZHvicgGEVkjIkuKZUvZEKp1XVCV44Z/r/d/0YXkPv3/oHXv8O9nGEbZUcxB6v8CbgV+nOP8ecBc73UScJu3NfrD54foLJc99uCWoY8jBMJuPOKhK+GeZRCscmXBsLethECld1yZvTxY6cqS5aEa1xUWqs0vLNEwjJKmaAJlD1lAAAAfjUlEQVShqs+KyOx+qlwA/NjLGPsnEYmIyBRV3VEsm8qKyoiLhmp8z03MGwrR2fDhb8Hm56G73c2AjnX07Lfvh6b08o78wi/FW2UvXO8EozLituF6t18Z7X0uVOeEzzCMkmI0w1ynAem5qLd6ZX0EQkQuBy4HmDlz5ogYNybwB6HhcGjdB01bhxbnP3mBe+VLItYjIN0dEGt32+52F6rb3ggdjdB+0G07GmHfRrftbM5xU3FeR0pIIlCZJiAV1YDPm2ckPd6JpO2nJr95+8lJRpkTxrJuk9eQVp5xn5Spklae5brkvr+ix7MKVpbGglKGMUhG81Obbapi1ml/qroSWAmwdOlSy0GdSXWD695pfM/N1i4mvoBrzEO1g782EYOOg+6VEpJGr6yxp+zge7Cz0XlGY2JyWx74Aj1dcr22Gd14qW4779gX6D0LPDXDOPnyeykt0o5TaS7SjjNTX0DajFv1ZuVqltm6uc73cw1K7xm+2v85st1P+9bP1jwMJiV91rqZZdr/6fSCPvfLYUu67dmuSZVpr03Pccb72yu1S+as6hwzqoc4IXc0BWIrMCPteDqwfZRsGfsEQjB+LjTvguYdDHeKfVHwBdyyqFUN+dVPxHsmEGr6F0wHvdWE0hmL094Vo707RjyeQHp9OXH1UVDcORRRRUTwCfhQRMDnOQ8+z7nwkb5VBPBrN75YJ754e5rH1d7b64q1u2685o7edWx+ShruTdZ0z7HnTMZO1rMDlw+U+6nP+RypNbLePouH2ee+/aTsSHm0GaI/0A8Ff0Xfa4bAaArEI8AXReSnuMHpgzb+UABqJ0G4zq1LHWsfbWuGh8/f0+U0BBIJaO2O0doZo60zTtynUIAo4cHgEwj4ffhFCPiFgM+H3ycEvJffLwTE1ztHX7Ibr7u955dhtl+KiTx/OSYyjpPimTXBYTKJXq4EiAJ4DU9q6+VO6pVQMSORYp860vt+3rEidCWErrjSEUvQFYfO7jjxHJ5CqokVEMTrVfS2yfMibj+17WmLk+Xp90q1zd5Oet3MB0tGo99jT88903szxbsm2Tvpk55niNdt6d4R6WVnEs1wRHp8D+113MvRSR08xmApmkCIyD3A6cB4EdkKXA8EAVT1duBR4MPABqANuKxYthxyBCthwlHOk+hvMaAyJJZQ2jrjtHY5YRhtPyqh0BXzusn6Gd/PFBK/r4KAL+yExBOWgE/KKtFpPAFdsbgTglicrliczlhiUP9nqQZSvSNNLzWGSzGjmC4Z4LwCpZ5WdOwi4tZ+CNW5sYl+02uPbbrjSkunE4SO7viYbB7yERIB53F4YpESDr/nkfh9BH2lqSDdcaUjFqc7lqDTe3XHR298KZZQmruUpk4lloCADyr8QsAHQR8EfULQ78p95aTKg8RCK8qdUI2bXNe0Fdr2jbY1BaMzlqA1KQqx0WtoMlFVDnQofh/UVkhBGxcFYnElFs894Ch4nohPCKaJid8vBNPEpFiouv+brliCTs8j6IzFSRRJtVWVjjg0dbrGvqmr77a5U2nqSnAwrU7bIJLl+sUTDT8EfELQ57YVnoAEfWnC4k+e7ykP+Hqu83vlfnH3c/cWrxyv67HnmoAXmBdXJ2rxBMQUYgmIJ9Qr945VvfKe+jFvf6habAJxKODzQWSmm3vQuGXMppJu74rT0hWjtTM+qr8+4wlld5uyvSXB9uYE21sS7GhJsK0lwc6WBB1e++0XiIbFe/kYFxailcK4sI9oWHodVwZ6+q2HgwLd8QTdcejoxxPJJVw6gP+VT9BQIbQgocreNmVHq3tvd7YkaMwhAt39fBSqg1BXIdSFhPqQMKPWT21IqKtwx3Uh1wh3x12D2p3AveKucXXH6p339nOc74hBc1eizz3i6urEvfJiiWUmPnGfwaTgDAUTiDInnlAS6l7qryEROQJt3Ip2HOiJHqRnIM1F6wg+X8+Anw9JRewMB1X35VDU23dbVSWBui+O0nMOd647nqC1M/dAZTHojCk7W12jv6MlkSYGyq7WBPE0Uyr8MLXGx5QaH8dPDjClxocq7O9IcKBd2d+hHOhIsLHReRfZGoiw3xOTyjTx8IQkXWQiYaEyMLz/CIURfS9z0RFTdrQkUiLg3mdlZ0uCna2JXg2/T3oa+roKYUqNj6OSx15Zr21IqK0orrc0VBJpv/S7vV/+3fEeIUl6BEkPQNV5Hn4fBMQJWvq+zwcBcR6H39v3Z+kaO/Ibg7fVBKKESCSUuGqqUY8l1JV55cmGM57o2U+kGtqe/USip/HNzgT8sQDB1u2Dio9Oftx8Ii6CTsQl85KerpSEa/FJQE/jnxSCob81RaG1W1MewPY0EdjRkmBPe29rq4NOBOZGfZw6I8C0WicI02p8jKvMvyspoe5X74EO99rfkXDb9p7jbc0J/rzb/ULORjgA0VCPgETSBCQpJpFwYT2ToZDsbkt6WE4M3PHOlgT7O3r/fVXeezy73sf7pjmhTQrvxCrBX+DGXoBQwI/PJ70+t4lEcT+zPnHdU/ihMmVJaWICUWBi8YTXV+ga9UQiraFPNv7er4Z4IuH6C9M+kCNFPBwlEawm2LIVX3d+k+uS5sVVIZ5eUvq0dyvrD8R5a3+ct73X9pbe9kfD7pfpokmBVOM/pVaYVuOjtkIK0tD6xDXekTDMGaBud1w5mBKTREpU0o+3tSRYu9eJTrb/jZDnmUSS4hFyAhIOSk/QD/RpDPse99w/PVAoOQUs+dntjMPOtG6hjrTfHwKMrxKm1vg4cYp7j3tEQAr2HmciQEXA1/vld6+BSHq9CdyPsHTPN+69SakfaWmecSKhvd6X5PuZeh+9f9LfP/X+Q5L76f8HyXBZnxcr60uG7JJxnB5Omx7eO8S31QSigOxp7mRXU8eINvTDQf0VdNUfNugxCcn5BypoAvEWtBcv7l68GHxRrzyRvbxQgtMVV95pTPDW/jhv7Yuz/kCC95p6BkonVPk4qqGCvzgiyIxaH1OrhSnVUBUsrf+4oF8YXyWMrwLof6JTPKE0pnkmjVkEZWdLgnV7nWcy3H7wVHx/xnHAB5OrXaN//OSAt++Ed1K1jwp/cX8tB/0+Quki4G2H2kCK14/v75lVcUhhAlEAYvEEWw+009wxRmfADnLluoHaliG3Pb0Eo0do3B2TgyBuUlVyZm08obzX2MHbu9tYv7uV9Xta2byvjZjXAtZXBpk7sY73HVnD3Em1HDGxhmhV9tlyHV66h57nx1JC1lv0Ymkil1aXBEG/j5pQAJ8P4nHnNcbiXpehd1wM/D6hoVJoqBy4btwbaM3VyPeaz5tlQlkpEPAJFQE/FQEh5PcT9La+IQ7GFofkZzbXtr869BwDPVkBGCBrAAOcGxwmEMOktTPGlgNtdMdK69fnmMSbhavexzLzHU2osqOxg/W7m1m/u4X1u5rZuLc1NX+gusLPERNruGDRNOZOrGHupBom1ITyb9hEQAI5n5+LioCPusog9eEAlQF6z1bOTPmhCWLxhGuk43EScTcfIJ5IEIvFXddkIk4skSAed90azphs1vSUSXrjkDqdpczbD6qmeXBpW9JyJhUVAfGjPr/bSgD1+UAC4PMhvgCBYBC/34/fX4E/6KfC7yMc9BMO+Ajk0T00JJugd+Pd6ziPOiUkooXABGIY7G7uYHdT55jpUhoLdMcTHGjrorGtm8a2Lg60dbPjoBOFjbtbaO1yndoVAR+HT6jh3HmTnRhMrGVKJDxik5qCAaG+Mkh9ZZCqisyvUe6vleDSCQSBcB7PSR+7iqeNZ8UyAhmS416JNI9lWJ/LDPHo2Y97YpTu7WnqHKjX4HsNv881+L5AkIDfj88fIBCoIBDwe1E3PZP9UilIitL4G0PBBGIIxOIJthxop2WsdimNML0b/W63394jAI1pgpAUgHQCPmF2QzWnHjmBuRNrOGJiLTPHVRU8qmUg+heF4uDzuTDj4BByrcUTvSPi4mkvTVePHG9jZp6hVHnO+q4bKtnQW4M/9jGBGCStnTHe299GLG5uQ3c8wd6WTnY3d7KnqZP9bV19fv03tnfR2pk9lLa6wk+kqoJIVZDZ46uJVkaIVAWJVFUQ9baRyiDR6gqCo9TIjIYoFAq/r/Chocahxdj6xI8yu5s62N186HQptXXF2N3UyZ6kCDS7v393Uyd7mjs50NbVp6e6qsJPNNnoN1QRrepp9CNVQXeu0h1XDHV6Z5EJ+HtEoTpkXxHj0MU+/XlQrl1KqsrWA+1s3t+Wavz3NDsx2N3c0eeXf8AnTKgNMaE2xJJZESbWhlPHE2tDjKuuIBQYm0uHmigYRl/smzAALZ0xtpRJl1JClc37Wlm7rYm12w/y+vYmDrb3zIGoDPqZ6DX4x0ypY6LX8DsBCBOpCpZVZstgQKgNB4mYKBhGVuxb0Q+7mzrY1TR202THE8q7e1tZu+1gShBaOp0XNKE2xJKZEeZPq+eICTVMrAtTUwaNpAi9M5n6haDflxosrfD3ZDQtpbh+wyhFxn6LUAS64wm27G/LObhaqsTiCTbsaUl5COt2NNHmRQVNqQ/zvsMamD+tjvlT65lYl0+QZekgQmr9g6C/Z+2DgNfgB9PEwDCMwmACkUFzRzdbD7SPiS6l7niCt3c1ex5CE2/ubKLDS4E5PVrJqXMnMG9qHcdNq6ehJjTK1g6MzwehgI9QwE9FoHfKBAuVNIyRxwTCQ1VTETqlSmcszls7ewThrZ3NdHnrIswaV8VZR09i/rR65k2ty5lOYrQRySECAZ/9+jeMEsMEgtLtUkqo8s6eVla9d4BX3zvAWzubiSUUAeZMqObc+ZOdIEypo65ycPmUiokIfTyAkJcqoVRDWw3D6MshLxDNHd1s2d9OfKSWeRqAlo4Yr245wKrNB1j13gEavbURD59QzfkLpzJvaj3HTq0rmQFlEais8FMbDlAZ9Kdlz7QBYMMY65RGKzMKqCq7vAlfo0m6l7Bq8wHe2tlEQqEmFGDxzAjHz4yyZGaUaHXpdBklw0NrQgFqQgGbrWsYZcohKRCqyjt7W2kbpS6lpJfw8uYDvJLhJVx8/AyWzopy5KTakml4RZxg1YSdIISHkhjIMIwxxyEpEPtbu0ZUHFJewub9zkvY1VzyXkIo6KPWE4TqigC+EhErwzBGjkNOIFSVPS3F71Zq7uhm9ZZGXt50gFe29HgJR0yo4WPHz+D4EvMSfD6oDQVTXoINJhuGccgJxIG27qIt7nOwvZvHX9/Jqk37e3kJS2ZGOH5WlMUzoyUVfipCKoFeVYXfBpYNw+jFISUQbq5DR1Hu/eK7+7nl6fU0tnVzxMQaPrZ0BktnRplbQl5CEr9PGF9TwbjqCpuAZhhGTg4pgSiG99DWFeNHz73Lr9/YxeyGKm74yDwOm1BT0GcUilDQx/iaEJHKoI0pGIYxIEUVCBE5F/hPwA/8UFVvyji/HPh3YJtXdKuq/rAYtqhqwUNa1247yM1Pvc3upk4+umQ6nzppZknOBq4O+RlfG6IuXDqT6QzDKH2KJhAi4ge+D3wI2Aq8JCKPqOobGVXvVdUvFsuOJI1t3anF7YdLVyzBT17YzMOvbmNSXZh/veg45k2tL8i9C4UI1FcGGV8TorLCwlINwxg8xfQgTgQ2qOo7ACLyU+ACIFMgik4yz1Ih2Linhe/85m3e29/GufMm85lT5pRUA+zzwbjqChqqQxaJZBjGsCimQEwDtqQdbwVOylLvoyJyKvA28GVV3ZKlzrA42D587yGeUB54ZSv3vPge9eEg13/kWJbOGlcgC4dPMCA0VLtV3UptUNwwjLFJMQUiWyuVOUL8C+AeVe0UkSuB/wbO7HMjkcuBywFmzpw5aEOG6z1sO9DOd598m7d2NfPBueO58tTDSyY5XmWFG3iurwxamKphGAWlmAKxFZiRdjwd2J5eQVX3pR3eAfxbthup6kpgJcDSpUsHFYbU2NZFZ/fQvAdV5dE/7+DOP26iwu/jmnOO4tQjJwzpXoWmNhxgfG2oZJL2GYZRfhSzdXkJmCsic3BRSsuAT6ZXEJEpqrrDOzwfWFdoI4bqPext6eQ/n1rP6i2NLJkZ4Utnzh31RXdEIFpdQUN1heVDMgyj6BRNIFQ1JiJfBJ7Ahbneqaqvi8gK4GVVfQT4koicD8SA/cDyQtpwsK170N6DqvK7t/dw++82Eksonz/9cM6dN3lUu298PhhfE6LBJrYZhjGCiGpprIOQL0uXLtWXX345r7rrdzWnluDMh4Pt3dz2zAb+sHEfR0+u5ctnH8nUSOVQTR02Ik4YxteYMBiGMTxEZJWqLh3MNWXbgX2wvXtQ4vDSpv3c8tv1NHfE+Jv3zeKixdNHLRpIBBpqKphQEzJhMAxj1ChbgdiTZ86lPqkyzp/HnPGjkypDxM1hmFAbKskZ2YZhHFqUpUAcbO+mvWtg7+H17Qf57pNvs6d5dFNlJAefJ5owGIZRQpSlQAzkPfRNlbGAY6fUjZB1PSTTbU+sDdusZ8MwSo6yE4imjv69h+54gusfWcva7U2cN38yl71/5FNlJPMkTawLEQpYuKphGKVJ2QnE7qbc8x5Ule/9dj1rtzfx5bOP5MyjJ46gZSYMhmGMLcpKIJo7umnvyr3W9E9f2sIzb+3h0yfNHHFxiFQFmVAbsgluhmGMGcpKIHb14z0889Zu/vfF9zjz6Il8fOmMnPUKTdJjMGEwDGOsUTYC0Z/38MaOJv7zqfXMn1rHF884YkRmRddVBphUFzZhMAxjzFI2ApEr59KOg+1881dvMLE2xFfPO6boYaQ14QCT68IltUaEYRjGUCgLgWjpjNHW2dd7aOmIccMv3kAVrv/IvKKm6A74hamRSupLJA24YRjGcCkLgdjd1HfeQ3c8wb8+to5dTR3c+Nfzi5pTqaGmgkl1YVuoxzCMsmLMC0RrZ4zWDO9BVfnBMxtYs+0g//ChI4u2XnQ46GNatJKqijH/NhqGYfRhzLds2cYeHli1lSfX7WbZCTM446jCh7OKwMS6EBNqQraKm2EYZcuYFojWzhgtHbFeZb9fv4cf/2kzpx05gU+eOPjlSQeiJhxgaiRsE90Mwyh7xrRAZHoPb+5s4rtPvs0xU+r40plzC/rrPuAXptSHiVRVFOyehmEYpcyYFYi2rt7ew66mDr75q3U0VIf4lw8fU9Dkd9HqIFPqK20Q2jCMQ4oxKxDpOZdaOmPc8Ms36E4k+H8fOa5goaahoI9pkUqqQ2P2bTIMwxgyY7Lla++K0+x5D7F4gn97/E22N7az4vx5zIhWDfv+IjCxNsSEWhuENgzj0GVMCsRub70HVeX2Z99h9ZZG/v7MuSyYHhn2vatDfqZGKi1FhmEYhzxjTiASCk3tznt46NVtPPH6Tj52/HTOPnbSsO7r97lB6Gi1DUIbhmHAGBSIWNwtBvT8xr381x83ccoR4/n0ybOGdc9IVZAp9WECttynYRhGijEnEPGEsn5XM9/+zdscOamWL589F98QxwkqAm4mdI0NQhuGYfRhzLWMsUSCb/zqDSKVQf7lL48Z0oQ1nw/G17iZ0D4LXTUMw8jKmBOI7Y0dzIgluPGvjyM6iElrAb9QGw5QVxmkNhSw6CTDMIwBGHMC0RVPcO25RzNz3MDhrBUBH3WVAerCQZvLYBiGMUjGXKs5oTbE4pnRnOcrK3zUhYPUVQYtVNUwDGMYjDmBqA/3niUtAlUVfuorg9SGgwVNsWEYhnEoU9TWVETOFZG3RGSDiFyb5XxIRO71zr8gIrPzu69b83nGuEqOmVLHYRNqaKgJmTgYhmEUkKJ5ECLiB74PfAjYCrwkIo+o6htp1f4WOKCqR4jIMuDfgE/0d9+KoI9jp9RZ9JFhGEaRKeZP7hOBDar6jqp2AT8FLsiocwHw397+A8BZMkB4kV/ExMEwDGMEKOYYxDRgS9rxVuCkXHVUNSYiB4EGYG96JRG5HLjcO2wRkbeKYjGMz3x2iTPW7IWxZ/NYsxfM5pFgrNkLcNRgLyimQGT7ma9DqIOqrgRWFsKo/hCRl1V1abGfUyjGmr0w9mwea/aC2TwSjDV7wdk82GuK2cW0FZiRdjwd2J6rjogEgHpgfxFtMgzDMPKkmALxEjBXROaISAWwDHgko84jwKXe/sXAb1W1jwdhGIZhjDxF62LyxhS+CDwB+IE7VfV1EVkBvKyqjwA/Av5HRDbgPIdlxbInT4rejVVgxpq9MPZsHmv2gtk8Eow1e2EINov9YDcMwzCyYTPLDMMwjKyYQBiGYRhZOSQFQkTuFJHdIrI2rWyciPxGRNZ729wZAUcBEZkhIk+LyDoReV1E/t4rL0m7RSQsIi+KyGuevTd45XO8tCrrvTQrJbfGq4j4ReRVEfmld1zSNovIJhH5s4isToYylurnAkBEIiLygIi86X2e31fi9h7lvbfJV5OIXF3iNn/Z+96tFZF7vO/joD/Hh6RAAP8FnJtRdi3wlKrOBZ7yjkuJGPCPqnoMcDLwBRE5ltK1uxM4U1UXAouAc0XkZFw6le969h7ApVspNf4eWJd2PBZsPkNVF6XF5pfq5wLgP4HHVfVoYCHuvS5Ze1X1Le+9XQQcD7QBD1GiNovINOBLwFJVnY8LEkqmMhrc51hVD8kXMBtYm3b8FjDF258CvDXaNg5g/89xea5K3m6gCngFN5N+LxDwyt8HPDHa9mXYOh33ZT8T+CVuMmep27wJGJ9RVpKfC6AOeBcvQKbU7c1i/znAH0rZZnoyVIzDRar+EviLoXyOD1UPIhuTVHUHgLedOMr25MTLersYeIESttvrqlkN7AZ+A2wEGlU15lXZivswlxI3A/8EJLzjBkrfZgV+LSKrvLQ0ULqfi8OAPcBdXjfeD0WkmtK1N5NlwD3efknarKrbgG8D7wE7gIPAKobwOTaBGGOISA3wM+BqVW0abXv6Q1Xj6tzy6bjkjcdkqzayVuVGRP4K2K2qq9KLs1QtGZs9TlHVJcB5uK7HU0fboH4IAEuA21R1MdBKiXTNDITXZ38+cP9o29If3ljIBcAcYCpQjftsZDLg59gEooddIjIFwNvuHmV7+iAiQZw43K2qD3rFJW+3qjYCz+DGTiJeWhXInn5lNDkFOF9ENuGyD5+J8yhK2WZUdbu33Y3rGz+R0v1cbAW2quoL3vEDOMEoVXvTOQ94RVV3ecelavPZwLuqukdVu4EHgfczhM+xCUQP6Wk/LsX18ZcMIiK4mefrVPU7aadK0m4RmSAiEW+/EvehXQc8jUurAiVkL4CqflVVp6vqbFxXwm9V9VOUsM0iUi0itcl9XB/5Wkr0c6GqO4EtIpLMLHoW8AYlam8Gl9DTvQSla/N7wMkiUuW1G8n3ePCf49EeUBmlQZx7cH1z3bhfNH+L62t+CljvbceNtp0ZNn8A5xKuAVZ7rw+Xqt3AAuBVz961wHVe+WHAi8AGnKseGm1bc9h/OvDLUrfZs+017/U68C9eeUl+LjzbFgEve5+Nh4FoKdvr2VwF7APq08pK1mbgBuBN77v3P0BoKJ9jS7VhGIZhZMW6mAzDMIysmEAYhmEYWTGBMAzDMLJiAmEYhmFkxQTCMAzDyIoJhDEmEJGGtGyaO0VkW9pxXtlVReSutPj7XHW+ICKfKpDNF3j2vSYib4jIZweof6aX0DDbuSki8mjavR7xymeIyL2FsNcwMrEwV2PMISJfB1pU9dsZ5YL7TCeyXjiCiEgIl5Ruqapu945nqerb/VxzI7BXVW/Ocu5HuFm83/eOF6jqmiKZbxiAeRDGGEdEjvBy3t+Oyxg7RURWisjLXj7869LqPicii0QkICKNInKT94v8eRGZ6NW5UUSuTqt/k7h1Ld4Skfd75dUi8jPv2nu8Zy3KMK0el8dpP4CqdibFQUQmiciD3nUvisjJInI48FngGs/reH/G/abgJnXi3W9N2t+/2tu/K82r2isi/+KVX+s9Z036+2EYA2ECYZQDxwI/UtXF6jJZXqtuXYSFwIe8dTMyqQd+p269iueBz+S4t6jqicA1QLJx/Ttgp3ftTbjMur1QlxfpCWCziPyviFwiIsnv2/eAb3k2fhz4oapuBH4I/Lu6tQf+mHHLW4H/FpHfisjXkjmAMp55mbrkiBfiUjv/WEQ+DMzEpVpfBLw/i/gYRlZMIIxyYKOqvpR2fImIvILzKI7BCUgm7ar6mLe/Crc+SDYezFLnA7hkfqhqMsVFH1R1OW7NjpdxGUtXeqfOBm73fvk/DES9fFU5UdVHgcNx+biOBV4VkYbMet597geuUtUtuNxM5+HSnrwCHAEc2d+zDCNJYOAqhlHytCZ3RGQubkW4E1W1UUR+AoSzXNOVth8n93ehM0udbCnAs+J1Ba0Rkf/FJSv8rHf9iaqabgNuCKXfe+0D7gbuFpHHcUKVKU53AD9V1afTbL1RVX+Ur82GkcQ8CKPcqAOagSavG+YvivCM53BdQ4jIcWTxUESkTnqvy7AI2OztPwl8Ia1ucvyiGajN9kAROSvpZYhIHS7X/3sZdf4eCGYM3j8B/K2X6RURmS4i4/P8O41DHPMgjHLjFVxq47XAO8AfivCMW3D9+2u8563FrdqVjgBfFZE7gHaghZ5xji8At4nIZbjv4NNe2c+B+0XkIuALGeMQJwC3ikg37ofdbar6qogckVbnK0BbctAauFVVfygiRwN/8jyUZuCTuDEKw+gXC3M1jEEibtGVgKp2eF1avwbmas9yjoZRFpgHYRiDpwZ4yhMKAa4wcTDKEfMgDMMwjKzYILVhGIaRFRMIwzAMIysmEIZhGEZWTCAMwzCMrJhAGIZhGFn5/wFvzp52Q8+IIgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f4580798b00>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_learning_curve(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see that by adding more model complexity, we've managed to lower the level of convergence to an rms error of 1.0!\n",
    "\n",
    "What if we get even more complex?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl8nGW5+P/PNUv2pEmbFtKNtlC27iv7jhxwQxahqEeKC4h4FH9HzlHP9wfK0d9Rjwc5isKvKBwXZFURPQrKoogg0Ja2FsrSFrovSdvsyazX94/7mWSSTpJJmyczyVzv12teM/PMMzNXkslzzf3c933doqoYY4wxvQVyHYAxxpj8ZAnCGGNMRpYgjDHGZGQJwhhjTEaWIIwxxmRkCcIYY0xGviUIESkRkZdEZK2IvCoiX82wT7GIPCgiG0XkRRGZ5lc8xhhjBsfPFkQEOFdV5wHzgQtF5ORe+3wcOKCqxwDfAb7pYzzGGGMGwbcEoU6rdzfsXXrPyrsY+LF3+xHgPBERv2IyxhiTvZCfLy4iQWAVcAzwfVV9sdcuk4BtAKoaF5EmYBzQ0Ot1rgWuBSgvL190/PHH+xl2Ydm7AcIlUDM915EYY3y0atWqBlUdP5jn+JogVDUBzBeRauBXIjJbVden7ZKptXBQ7Q9VXQGsAFi8eLGuXLnSl3gL0vcWw4Tj4cqf5ToSY4yPRGTLYJ8zLKOYVLUR+BNwYa+HtgNTAEQkBIwB9g9HTMYYY/rn5yim8V7LAREpBc4HXu+122PA1d7ty4Gn1aoHDj/7lRtjMvDzFFMd8GOvHyIAPKSqvxWRW4GVqvoY8CPgpyKyEddyWOZjPCYTGxNgjOmDbwlCVdcBCzJsvzntdifwQb9iMMYcLBaLsX37djo7O3MdivFBSUkJkydPJhwOH/Zr+dpJbYzJP9u3b6eyspJp06Zho8pHF1Vl3759bN++nenTD39kopXaKHh2gCg0nZ2djBs3zpLDKCQijBs3bshah5YgjClAlhxGr6H821qCMMYYk5ElCGPDXM2wamxs5Ac/+MEhPffd7343jY2N/e5z88038+STTx7S65ueLEEUOjvVYIZZfwkikUj0+9zf/e53VFdX97vPrbfeyvnnn3/I8flloJ8tH1mCMMYMqy9+8Yts2rSJ+fPnc9NNN/GnP/2Jc845hw996EPMmTMHgA984AMsWrSIWbNmsWLFiq7nTps2jYaGBt555x1OOOEEPvnJTzJr1iwuuOACOjo6AFi+fDmPPPJI1/633HILCxcuZM6cObz+upurW19fz7ve9S4WLlzIddddx1FHHUVDQ48ScCQSCZYvX87s2bOZM2cO3/nOdwDYuHEj559/PvPmzWPhwoVs2rQJVeWmm27q2vfBBx8EyPiz/exnP2Pp0qXMnz+f6667jkQi0ed75ZoNczVkKH9lCsRXf/Mqr+1sHtLXPHFiFbe8b1afj3/jG99g/fr1rFmzBnAH0Zdeeon169d3Dc285557GDt2LB0dHSxZsoTLLruMcePG9Xidt956i/vvv5+7776bK664gl/84hd85CMfOej9amtrWb16NT/4wQ/49re/zQ9/+EO++tWvcu655/KlL32Jxx9/vEcSSlmzZg07duxg/XpXPi51auvDH/4wX/ziF7nkkkvo7OwkmUzyy1/+kjVr1rB27VoaGhpYsmQJZ555JkCPn23Dhg08+OCD/PWvfyUcDvPpT3+a++67j1mzZmV8r1yzFkTBs1NMJveWLl3aY9z+d7/7XebNm8fJJ5/Mtm3beOuttw56zvTp05k/fz4AixYt4p133sn42pdeeulB+zz33HMsW+YKN1x44YXU1NQc9LwZM2awefNm/umf/onHH3+cqqoqWlpa2LFjB5dccgngJqWVlZXx3HPPcdVVVxEMBjniiCM466yzePnllw/62Z566ilWrVrFkiVLmD9/Pk899RSbN2/O+F75wFoQxhSw/r7pD6fy8vKu23/605948skneeGFFygrK+Pss8/OOK6/uLi463YwGOw6xdTXfsFgkHg8DrgJZQOpqalh7dq1PPHEE3z/+9/noYce4vbbb8+4b3+vl/6zqSpXX301//Ef/3HQfr3f65577hkwRr9ZC8IYM6wqKytpaWnp8/GmpiZqamooKyvj9ddf529/+9uQx3D66afz0EMPAfCHP/yBAwcOHLRPQ0MDyWSSyy67jH//939n9erVVFVVMXnyZB599FEAIpEI7e3tnHnmmTz44IMkEgnq6+t59tlnWbp06UGved555/HII4+wd+9eAPbv38+WLVsyvlc+sBaEsWGuZliNGzeO0047jdmzZ3PRRRfxnve8p8fjF154IXfddRdz587luOOO4+STe69UfPhuueUWrrrqKh588EHOOuss6urqqKys7LHPjh07uOaaa0gmkwBd3/p/+tOfct1113HzzTcTDod5+OGHueSSS3jhhReYN28eIsK3vvUtjjzyyK5O8ZQTTzyRr33ta1xwwQUkk0nC4TDf//73KS0tzfheuSYjrbq2LRg0xH5wClQfBR96INeRmGGyYcMGTjjhhFyHkVORSIRgMEgoFOKFF17g+uuv7+o0Hw0y/Y1FZJWqLh7M61gLwhhTcLZu3coVV1xBMpmkqKiIu+++O9ch5SVLEAZI5joAY4bVzJkzeeWVV3IdRt6zTupCZzOpjTF9sARhjDEmI0sQxhhjMrIEYazShjEmI0sQBc/6IEz+q6ioAGDnzp1cfvnlGfc5++yzGWgI/O233057e3vX/WzKhxcySxAGdOSVITaFaeLEiV2VWg9F7wSRTfnwXMiX0uCWIAwk4rmOwBSQf/3Xf+2xHsRXvvIV/uu//ovW1lbOO++8rtLcv/71rw967jvvvMPs2bMB6OjoYNmyZcydO5crr7yyRy2m66+/nsWLFzNr1ixuueUWwBUA3LlzJ+eccw7nnHMO0F0+HOC2225j9uzZzJ49u6vmUn9lxdM9/PDDzJ49m3nz5nVVcU0kEnzhC19gzpw5zJ07l+9973uAK9i3YMEC5syZw8c+9jEikUhXLLfeeiunn346Dz/8MJs2beLCCy9k0aJFnHHGGV2zsjO9l19sHkTBE1BLEAXr91+E3X8f2tc8cg5c9I0+H162bBk33ngjn/70pwF46KGHePzxxykpKeFXv/oVVVVVNDQ0cPLJJ/P+97+/zzWW77zzTsrKyli3bh3r1q1j4cKFXY99/etfZ+zYsSQSCc477zzWrVvHZz/7WW677TaeeeYZamtre7zWqlWruPfee3nxxRdRVU466STOOussampqsiorfuutt/LEE08wadKkrlNWK1as4O233+aVV14hFAqxf/9+Ojs7Wb58OU899RTHHnssH/3oR7nzzju58cYbAVcd9rnnngNc3aa77rqLmTNn8uKLL/LpT3+ap59+OuN7+cVaEAaS+dGcNYVhwYIF7N27l507d7J27VpqamqYOnUqqsqXv/xl5s6dy/nnn8+OHTvYs2dPn6/z7LPPdh2o586dy9y5c7see+ihh1i4cCELFizg1Vdf5bXXXus3pueee45LLrmE8vJyKioquPTSS/nLX/4CZFdW/LTTTmP58uXcfffdXaeHnnzyST71qU8RCrnv4WPHjuWNN95g+vTpHHvssQBcffXVPPvss12vc+WVVwLQ2trK888/zwc/+MGuhYV27drV53v5xVoQhU5wxfpinRAuyXU0Zrj1803fT5dffjmPPPIIu3fv7lqX4b777qO+vp5Vq1YRDoeZNm1axjLf6TK1Lt5++22+/e1v8/LLL1NTU8Py5csHfJ3+atJlU1b8rrvu4sUXX+R///d/mT9/PmvWrEFVD4pvoNp3qdLgyWSS6urqjPWhMr1X78WUhoq1IAygEG3NdRCmgCxbtowHHniARx55pGtUUlNTExMmTCAcDvPMM8+wZcuWfl/jzDPP5L777gNg/fr1rFu3DoDm5mbKy8sZM2YMe/bs4fe//33Xc/oqNX7mmWfy6KOP0t7eTltbG7/61a8444wzsv55Nm3axEknncStt95KbW0t27Zt44ILLuCuu+7qWoNi//79HH/88bzzzjts3LgRcJVhzzrrrINer6qqiunTp/Pwww8DLrGsXbu2z/fyi7UgCp73DSfSAuW1/e9qzBCZNWsWLS0tTJo0ibq6OsAt5fm+972PxYsXM3/+fI4//vh+X+P666/nmmuuYe7cucyfP79r/YV58+axYMECZs2axYwZMzjttNO6nnPttddy0UUXUVdXxzPPPNO1feHChSxfvrzrNT7xiU+wYMGCPlep6+2mm27irbfeQlU577zzmDdvHrNnz+bNN99k7ty5hMNhPvnJT/KZz3yGe++9lw9+8IPE43GWLFnCpz71qYyved9993H99dfzta99jVgsxrJly5g3b17G9/KLb+W+RWQK8BPgSFw1uBWq+t+99jkb+DXwtrfpl6p6a3+va+W+h9hdZ0C4FN5zGxw5O9fRmGFg5b5Hv5FQ7jsO/LOqrhaRSmCViPxRVXv3Fv1FVd/rYxxmQArJmPVDGGN68K0PQlV3qepq73YLsAGY5Nf7mUOU3okWbctdHMaYvDMsndQiMg1YALyY4eFTRGStiPxeRPJjBfVCFe17nWAzuoy0lSRN9obyb+t7ghCRCuAXwI2q2tzr4dXAUao6D/ge8Ggfr3GtiKwUkZX19fX+BlyIUh8oa0EUhJKSEvbt22dJYhRSVfbt20dJydCcKvZ1FJOIhHHJ4T5V/WXvx9MThqr+TkR+ICK1qtrQa78VwApwndR+xlzQElGIRyBUPPC+ZsSaPHky27dvx75sjU4lJSVMnjx5SF7LtwQhbobIj4ANqnpbH/scCexRVRWRpbgWzT6/YjKZ9JpoFG2zBDHKhcNhpk+fnuswzAjgZwviNOAfgb+LSGo64JeBqQCqehdwOXC9iMSBDmCZWrs3tyItUDY211EYY/KAbwlCVZ9jgMUGVPUO4A6/YjCHwGZUG2M8Vmqj0PWuZZOIQjyam1iMMXnFEoQ5mLUijDFYgjDQPcw1xRKEMQZLECZTN1HEEoQxxhKEySQRgUQs11EYY3LMEoTJLGJlN4wpdJYgDJBh6omV3TCm4FmCKHR9LAhvHdXGGEsQJrN4p/VDGFPgLEEYMp5iAmtFGFPgLEEUvH6qodhwV2MKmiUI0zfrqDamoFmCMH2Ld0AinusojDE5YgnC9NkFAVg/hDEFzBJEoetrmGuKJQhjCpYlCNM/66g2pmBZgjD0e44p3gHJxPCFYozJG5YgCt4Ap5jA6jIZU6AsQZiB2XBXYwqSJQgzMOuoNqYgWYIwB68o11vM+iGMKUSWIArdQMNcAVBrRRhTgCxBmOxYP4QxBccShKH/qdQemw9hTMGxBFHwsjnFBMTaIZn0NxRjTF6xBFHosswP1g9hTOGxBGGyZwnCmIJiCcJkzzqqjSkoviUIEZkiIs+IyAYReVVEPpdhHxGR74rIRhFZJyIL/YrH9CXrc0wuQVg/hDEFw88WRBz4Z1U9ATgZuEFETuy1z0XATO9yLXCnj/GYw6YQs1aEMYXCtwShqrtUdbV3uwXYAEzqtdvFwE/U+RtQLSJ1fsVk+jDQTOp0NtzVmIIxLH0QIjINWAC82OuhScC2tPvbOTiJICLXishKEVlZX1/vV5iFKauZ1GmsH8KYguF7ghCRCuAXwI2q2tz74QxPOejrrKquUNXFqrp4/PjxfoRpshVrG1yLwxgzYvmaIEQkjEsO96nqLzPssh2YknZ/MrDTz5hMJoM44GvSWhHGFAg/RzEJ8CNgg6re1sdujwEf9UYznQw0qeouv2IymQzyFBPYfAhjCkTIx9c+DfhH4O8issbb9mVgKoCq3gX8Dng3sBFoB67xMR6T0SEkiEgrVA59JMaY/OJbglDV5xjg6KOqCtzgVwzGJ6l+iMF2cBtjRhSbSV3ogiHobBpcx7P1QxhTECxBFLrj3g2NW2HXmoH3TWcJwphRzxJEoZvzQSitgbX3D+551lFtzKhnCaLQhUpg9mWw7SXYtyn750VbbT6EMaOcJQgDJ17sEsXaB7J/jibdIkLGmFHLEoSB4ko44X2w6Slo3ZP986wfwphRzRKEceZcDgj8/ZHsnxNp8S0cY0zuWYIodKm5DBVHwDHnwYbfZH/gj1pdJmNGs34ThIgEReQ/hysYk2PzlkG8E177dXb7awJiHf7GZIzJmX4ThKomgEVeXSUz2o2dAVNOgvW/gHgku+fYcFdjRq1sTjG9AvxaRP5RRC5NXfwOzOTIvGXQcQDe+kN2+1uCMGbUyqYW01hgH3Bu2jYFMpXvNiNOr8Zh3XwYfxyse9DNsg4E+3+6rTBnzKg1YIJQVauwWkhEYN5V8ORXYMtfYfqZ/e+f6ocIlw5LeMaY4TPgKSYRmSwivxKRvSKyR0R+ISKThyM4kyPTzoDKibDm/uxGKVkrwphRKZs+iHtxC/tMxK0X/RtvmxmtAkGYdyXUb4Dd6wbePzpC50PYCCxj+pVNghivqveqaty7/A9gC0OPdsdeCCXV2RXxG4kzqpt3Qv0bEOvMdSTG5K1sEkSDiHzEmxMRFJGP4DqtzWjQ1wjmUDHMvhS2/g32b+7/NZLxkfVtvHGbV1JEoWlbrqMxJm9lkyA+BlwB7AZ2AZd728xolyrit+7BgfcdCa0IVdj/NrQ3dG+LtkJbQ9/PMaaADTiTGrhMVd+vquNVdYKqfkBVtwxTfCaXSsbA8e+Bt56E1r3975vvdZmSSdcS6mw8+LHmnRCPDn9MxuS5bGZSXzxMsZh8NOdyQN3s6v5EWvK3FZFMwL6NEGnO/Lgm7FSTMRlkc4rpryJyh4icISILUxffIzP5obIOjj534CJ+moCGt6B5V34V8EvEXFyxAZJXpBna9w9PTMaMENnMpD7Vu741bZvSc2a1GbGyKLM190rY+KRLEvM/1M+OCq273Xn96qmuozuX4hG3Sl4iy7pSzTuguAqC2fxbGDP69fufICIB4E5VfWiY4jH5qHYmTF7i1oqYczkEi/rfP9rqhpCOmQxlY4cnxt5iHS45JGPZPycZh+btUDPNt7CMGUkG6oNIAp8ZplhMPpt7JXTsh7f+mN3+moDGLW7UUDLhb2y9RVrdaaXBJIeUjgPQ2TT0MRkzAmXTB/FHEfmCiEwRkbGpi++RmfwyaRGMm+nWrdZk9s/rbIT614dvlFNnE+zf5BLUoWra7kY9GVPgsp0HcQPwLLDKu6z0MygzjLJd6kPElQJv2gZbnh/ceySibhRR805/O7Db97sWy2ASWCaJKLTsHJqYjBnBBkwQqjo9w2XGcARn8syMs6DySNeKOBSte6DhTX/KW7Q1uFNaDFECaqu3IoSm4GVTzbVMRP6PiKzw7s8Ukfdm8bx7vAqw6/t4/GwRaRKRNd7l5sGHb4ZVIARzroA962H33w/tNWLt0PAGtA1htZaW3f7MY2jall9Ddo0ZZtlWc43SPdx1O/C1LJ73P8CFA+zzF1Wd711uHWBfkw+Ou8gNBT3UVgS4U0BNW93M5kT88OJp2g4tuw7vNfoS73TJx5gClU2COFpVvwXEAFS1gywGz6vqs4DNPBptwqUw6xK3mNCBw6y40tnkOrA7+5jh3B9V9/5t9YcXw0Ba94ysQoTGDKFsEkRURErxTu6KyNFAljOPBnSKiKwVkd+LyKy+dhKRa0VkpYisrK/3+YBQcLLspE436xIIFmdXxG8gyZgbddS0I/vTOam6Sh3D8f1DXfVXO9VkClA2CeIW4HFgiojcBzwF/MsQvPdq4ChVnQd8D3i0rx1VdYWqLlbVxePH21IUOVda7U41vfWHoauE2rY3u/UZkgmXUPqqq+SHWJv/LRVj8lA2o5j+CFwKLAfuBxar6p8O941VtVlVW73bvwPCIlJ7uK9rhsncK1xfwkBF/AYj3uE6sFv7OBgn4m64bDQHo4tadrnSHcYUkGxaEKjqPlX9X1X9raoOyVdGETlSxA3CF5GlXiy2ENFIUTURpp8Frz02tFVcNenKXezb5ArtpcSj3hDZ9qF7r8HG1WgVX01hySpBHAoRuR94AThORLaLyMdF5FMi8ilvl8uB9SKyFvgusEzVTvSOKPOWudMvG34z9K8dafY6sJvcaaeGN7MvuueXaMvQDs81Js/5VrZSVa8a4PE7gDv8en+TpWxnUmcy/jiYtNAV8Zt9GQTDQxcXuOJ5+zeDBA5/dvRQad4BJVVD/7Mak4eymSh3tIgUe7fPFpHPiki1/6GZEWHuMreE58Yn/XuPfEkO4C0utD3XURgzLLI5xfQLICEixwA/AqYDP/c1KjNyTF4CY492Q17z6UDup85G6MiwdKkxo0w2CSKpqnHgEuB2Vf08UOdvWGbESBXxO/COmzxXKJq2D38Zc2OGWTYJIiYiVwFXA7/1ttkJ2FHjMPogUo4+B6omwZNfhZdWFMbM42TM9UcYM4plkyCuAU4Bvq6qb4vIdOBn/oZlRpRACC6+A445D9b8HB5eDu/8ZfTPPm7fN3zrXBiTAzLSRpYuXrxYV6605SiGTDIJu9cO3evtWgd/vd2NPppyMpz2WTdnIt+ougN8+WHOzQwWw/jjIeDbiHFjhoSIrFLVxYN5TjajmN4rIq+IyH4RaRaRFhEZxjoHZkSpmwuXroCTb3CJ5+GrYdWP82cWcjIBm/8Mj14P910OK+89vNdLRPyrJmtMjmUzD+J2XKmNv9tENpOVQAjmfhCOPhte+AGsuhfeegJO+xxMOSk3McUjrnbUugddB3PVJNfCWf1j9/ii5Yc+J6StHkproKhsyMI1Jh9kkyC2AestOYxShzNRbiDl4+H8W2D7e+Cv/w2//1eYdiac+hmomODf+6aLtMBrv3Y1ozoOuMl9538Fpp3hfvZnv+0lCYVF1xzi70Ohcat7bT9/n8YMs2wSxL8AvxORP5NW5ltVb/MtKjO6TF4Ml/8I1j0Mq38CD70ECz8Kcz7o34zk1r1uhvfrv3GjqqYshXlXQd38ngfxM7/grlf/xPVLLP7YoR3k4x3uPSuPGJr4jckD2SSIrwOtQAlQ5G84ZtQKFsGCD8Mx58Lzd7jhsG8+AaffCBMXDN377H8b1j0Abz0JKBx9rpunMe6YzPtLwCUJEXjlp+45iz9+aEmiZReUjIFwyeH8BMbkjWwSxFhVvcD3SExhqKyDf/g6bH0B/vpd+O3n4Zjz4eTroWzcob2mqlsje+397nVDJTDrA66FUnnkwM+XAJzxz4DAKz9zS2MtOZQkoW4d69qZh/BDGJN/skkQT4rIBar6B9+jMcMvV+fMp54CExe6eRNrfg5bXoAlH4MTL3ad3NnQpJu9vfYB2POq+/a+6BqXHErGDC4eCcAZ/w8gsOZngMKSTwz+9xNtdYsoHe7wWWPywIDzIESkBSjH9T/EcFNvVVWr/A/vYDYPwgc7X8nt+zdtd53Y2192p4JO/zwc0ecKtJCIwlt/dImhaZtrlcy9wq1yFzrM0zuahL98x/VdzP8QLPnk4JOEBKCowruUu4t1XpscO5R5EP1+VfMW9JmlqlsPKzJj+jNmMlz0LXj7WXjhDvj1DXDcu+Gka6EkrXBwtBVe+w2sf8RNchs3E867GaafmX2rYyASgDM+774Grfm5O3219NrBHeA16dazSC2LKgEIl3Uni6IKCASHJl5jfNTvf5Wqqoj8Clg0TPGYQiUCM86CKUtg9U9h3UPwznOw9JNu7sSrv3Kr18XaYNIiOPtL7tqPb+YScK0YxPVrwOCTRDpNuuSWvlRqqLRnwgjZ+A+Tf7L52vU3EVmiqi/7Ho0x4TI46To49h/gue/AX/7LbZcAzDjbjUiqPdb/OCTgRljJECWJ3uId7tLureAbLOpOFkXlEC4dmvcx5jBkkyDOAa4TkS1AG919EHN9jcwUtppp8N7bYdPTcOBtd8ppuGs6SQBOu9HdXnu/O9100nX+tFoSUeiIusl8ABJ0iSJY5OaKBELuEgxDIHU/x/WfVN2qf+kXVfd7S4/ZTqeNWNkkiIt8j8LkVsWR7nx5rD3XkfQk4irE5jqG024ExM2vQOGkT/nf6ayJ7j6MPmMLZkgeIZdAgqHuRBIMZxdvMukd6GPedeLgBJBITwbZrochByeMQK/7llDy0oAJQlW3DEcgJoeq6oA698+f6lyNtLiDgPGSxOfc7XUP4pLE9bkfmaQJiGd5kO46+HrJAw5OAL6tCKiuhZSIZrl/Hwml6/ftXYv0ut37sUz3B3iMTPsdxvb09zoo3rTbuf4s9WGIhn6YUSEYgrKx7gIQbXeJItIM0TbcDLIClUoSIq4DXXGT+/L0H/sgqSRAZ64jycJgE8po0iuBZPp8DVgWb+j+Ty1BmL4VlblL5RHu22YqWURaCvOfVwRO/Swg8PeHAIWTPz1ykoQZAbyDeyoJ5Pg7mSUIk51AEEqr3QUg1tmdLKKtPp6eyDMicOo/ueu/P+z+kU+5wZKEGZUsQZhDEy5xl4oJrnMz2urWRRioY3U0EIFTPuNur38EUHffkoQZZSxBmMMXCEBJlbtE211V09GeKLqShHhJAksSZtSxBGGGVlEZjDvadWq37B7diULEnV4CSxJmVLIEYfxRVF4YiSKVJNL7JFJ9FMaMcL5NxRSRe0Rkr4is7+NxEZHvishGEVknIgv9isXkUCpR1B4LxTkpAOw/ETeaac4V8Oov4fnvZTEU0Zj852cL4n+AO4Cf9PH4RcBM73IScKd3bUaj0d6iEPHmReDmSWx9wVWprZroypGnXxeV5zpaY7LiW4JQ1WdFZFo/u1wM/ETdghR/E5FqEalT1V1+xWTyQCpRRFqhdc/oShQiboZ11STYuQZadsKmNw7+GYur3D5VdVA50V2nEkj5+MIqNaFJN1S6swk6m73rtEukuftaAl5tKu8SKup5v/e2UHH/j6f2Ee/3Lb0mqSHevLX0CWt9TWTLNFO6v5+7dwtT+35MxP3sOZDLPohJwLa0+9u9bQclCBG5FrgWYOrUqcMSnPFZcYW7jLZEIeJWxTvx4u5tkRbXamreAc27XOJo3gX1r8PmP/esaRQIuWVSK+vcpbgCgsXdB7tQUdr9Yne/63aGffw+sGjSXZK7fIT1AAAeAUlEQVQJd52IdR/UB7qk5tH0NYcmGHbrgZRUuaSq6lqgif0Qj3bPtu66xPz9WXNJAl7JkVCvEiRZ3JfQIa+XkssEkSnFZjxxq6orgBXgVpTzMygzzNITRctuiLbkOqKhV1zpLpnWqk7GobW+O2mkX9e/AbEOVzzvUAXCPRNGKolI0CUmTbh5LKkDvSZ6HvB73+66nxjc5MhA2C0Dm7qMO7rn/ZIxLgmk3w+VDH6hpkSsO2FkSiLxKCQiPW9r0vvWrt4RKHW716xmeu2Tek7XPpn27yf+vmo69X5Mk30UTxzgfjwKyfa0MiuDl8sEsR2YknZ/MrAzR7GYXCuugOJjRneiyCQQ8k4x1bn2cybJhHeAi7h/+njEu5060KXdHnAf73GS7lupBL3rQPe31NS2QODgfQKp+8G0x9P2CYZ6HfS963Cp/yO7JNCdBM3Bbhz8OKBcJojHgM+IyAO4zukm638w3YmiBVr2FE6i6E8gCIFSW0TIDDvfEoSI3A+cDdSKyHbgFiAMoKp3Ab8D3g1sBNqBa/yKxYxAqdMykRboaMSXqmWadOfCC6WOlDGD5OcopqsGeFyBG/x6fzNKpBKFXxJxt+xnW8Phnes3ZhSymdSmsAW9UUMVR7jlPtvq829lvQIXSyixRBJFEaS7O1dw93ut2SPe8NTuAajuOXm8Lk/esgRhDLgjR2qxpEiLSxSdTQM+LZmEhCqJpJLQJMkkxJPudFhAhIA3ND4gEEAQcQe0QOoayfnS0vkgkYRYIkkskSSaSBKNJ4knkkTiSV+WROhKHqkkE/D+Xt7G1N9OvETj/l6pbZK2zeu7p+ffVUQIIARH+N/WEoQxaRJJJR4sI1ExlXhRJ8mWvSTb9pFIJkgk1SWEpJJQJZ4YuoNX1wEn7UAlIgQD3ZeACKFAz22hwMj5SqwKsaRLALG4Eo0nuloHqaQ6bLGkxeRWsEvfOnRSf9dgsPvvFxAIpP6GEiAQgJAIgYC7hETypqVjCcIUHFWlJRKnuSNGLOEO9PGkawUcXEJpHBRVE+rcRzC6H0n6s5Ke4loigz1QpQ5AIe8AlEoaqYQSDErXwScUCAzLN9pYUonHk0QTSjSRIJ5QLyn40xoYrHhSaYkqzRF3SQIlQaE4BMVBocS7Lg5C8DATcOrvmogP7icPCAQDAYJeawS6r6FnC6i3jNsG9e7dLEGYgtEejdPYHqOxPUZiMN9YA0HiZROIl44nGGki1NmAxPOjn2KwByDBHfSkj6+oOtAhvJ+HVd3zh7YxECAZKiUZdvWrxJugJ5pAkwki8QQtnXFaOhI0RxM0RxI0R5SmiNIcVVoiSnM02ZUMmqNK2yDGIoQDUByE4pD0SB69r4tDXpLptW95WNylSKgIC+Vhty0c7P+QnVRIJpLketiEJQgzqnXGEjR1uKQQjR/mcFYREiXVJEqqCcTaCHXUE4iOrBIhSqqPJB++y2cgQZKhcjqkhJ0dIXa0wu7mCLua22hqj9LSGae5M+aSQmecaKLvv2lZUZDK4iBVJUVUlgWpqwlSVRzwLkJVUYCqYgiK0hlXInElkvCu40k6E0okpu66x+NJOuNKcyxtX++xaCK732tRUCgvClAWDlBRFKCsKEh5UcBdwkHKioOUF7lLcUgQVcD9rKJJpGtWt5sFLiiSGq6tSUC79zkMliDMqBNLJL2kEKUj6s8ch2S4nGi4HElECHXsIxg50LOmkhlQe0zZ2RZge0eIHW0BdrYqO5uj7G7eS0Nrz1N55cVBasqKqCwJc0RVCcdMCFFZEqaqJExlSYiqEnff3Q5TURIinIMe4qQq0bjrXO+IJWiPxGmLJmiPxmmLJGiLxN3t6MGP1TfFaY9EaYvGiRzul5khYgnCjAqJpNLcEaOxI0ZbJD5syzFosJhYxUTai8ezc/du3t69j00HonTE1HVGCgS9judgajRT2iXojYw5+DE3AqYkCKVhoSwklIahNJS6LZSFDv8cud+aI8rO1iQ7W5LsbE2yow12tsLOljiNnT0TanVZmLoxpcydVE1ddQl1Y0qpG1NC3ZgSKkvCvsSXGnmU1Ez9T4MXEKEkHKQkHGRM6aHHHE8kaY8maIvGD2r5Zorz4E09t6jCB745+DgsQZgRK9XZ3NgWo7kzNmxJoTOW4J2GNjY1tLGpvpXN9a1s2dfeNRKnJOROGyTVnY9PXSeS3fcTypCcqw8HoCwslIYDlIWEkrTbpeHAQZeycICScJBwAJJekb6kJtGkuySTiqa24WJUTfV1uNtJr68hdTup7uSHKnTGXULY1ZpkZ6vrDE5XW1FE3ZhSls7omQCOHFNCWZF/hyMRKAkHKA4FKfauU/fTJZPa4++mXX8vdTXztPtx7fH37d63+7lKItnz8Ww/o6FggKrSAFWHkWSGgiUIM+K0ReI0dsRoGmxn8yFo7YyzqcElgU31bWyub2VHY0fXwb2yJMTR4yu4eP5Ejh5fwYzaCuqqSwhkOU4x6c2hSKobQtt1AEoqiWSSzliCzrjSEUvQEU246/TbGbY1RxPs7UjQEYt3bRuuUaQBgQkVIeqqyji9rpyJ1WVdrYEjqooPOiAPtd6JoCQcpDh0cCLoM/6Am7/gF+36e3cnoEQqgSS7k1HvhNM7oh6TA3s92vOxw2MJwowIQ9rZnIGqsr8tyuaGtq5ksKm+lb0tka59aiuKOXp8OacfU8vRE1wyqK0o6nNEUDYCIgQGGNFyuFTdMNP0pJJIao+JX10TwLzTXt2Tvci4X49rb5JY19DaYTjtFQjgkkDISwLhACWhIEWh/J6ZJt6Q5JHCEoTJG8mkN14+kSTuTaCKJty36KHubG6PxlmzrZGNe72WQUMrje3dgwonjinhuCMruWh2HUePL2fG+IrDOqecSyLiHUyDVOc6mEEQgXAwQDgoFIV6nhbK90QwWliCMMMmVUohFu9OBF3lFeLq++mipo4YL729j+c37WPNtkbiSSUYEKbUlLJoag0zxldw9PhypteW+3o+3HQLeQf/omCg6zqcug72PV/DDA/7LzBDIpHs/sYfiye7SiiktwiGqxM53b7WCH/bvI/nN+9j/Y4mkgoTKot579yJnDxjLDMnVNq3UR8FAz0TQKo1kNpmCSC/WYIwfUod9BNJdad8ksmubfGEEk+6EhW5Ovj3ZXdzJy9sauD5Tft4fbdbcGhyTSmXL5rCqUePY0Ztua8HppJwgFAw0F1BNFPFUZEej3dthx5VSlOzk4Gu1S67VsL0RhK5bW5j71Uy0/fB2y91272/pN1Obc8cb/p+B8XsPRiQtIQQDBDI82G4pn+WIApYIqm0RuJ0xhLegd59848nc/eN/1Bt29/O85saeH7zPjbXtwEwY3w5HzlpKqceXcuUsWW+vbeIm7VbVeomavk9UseY4WIJosB0xhI0d8Zo7YzTHk2MqCSQTlXZVN/GC5v38fymBrYf6ADg+CMr+dhp0zhlRi1Hjinx7f0DAbpm8VaWhPN+wpoxh8ISxCiXSCqtXv2a1kiceJa1YvJRUpU3dre4lsKmfextiRAQmD1pDO+dU8fJM8YxrsK/BeuLQgFXyqE0THlR0M6fm1HPEsQo1BFN0NIZoyXiJkqN1FYCQCSe4PVdLTy/eR9/27SP/e1RQgFh/pRqli2ZwtLp43wdflpaFKTKSwolYTt1ZAqLJYhRIJ5I0hqJd1W49Hu4qB9aOmNsO9DB9gPtbNvvrrcf6GBPcycKFIcCLDqqhlNmjGPJtLGUF/vz0RWBiuJQV39CLgq+GZMvLEGMUO3R7oTQER0ZVUSTqjS0RNh+oINtXgJIXTd1dE9SCweFSdWlzDyignOOG8+M8RXMn1Lt2zf4UFC6+hIqi0M28sYYjyWIESKWSNLqJYTWSH63EmKJJDsbO9ieahEc6G4RpJcxriwOMXlsGUunj2VKTSmTa8qYUlPG+Mpi3zt9S8IBVy66NGST4ozpg/1n5KlIPJFWPz7hS/2hoRBPJHljTwtrtjXydkMb2/a3s7u5s0dxuPGVxUypKWXWxDFMrillSk0Zk2tKGVMaHraO3kDAnTpKrRlgp46MGZgliDzR4dV+b4+463wdbaSqbDvQwZptB1izrZH1O5rpiCUICEyqKWNabTlnzBzPZK9FMLmmNGeduyXhABXeqSMbdWTM4FmCyAFV7VoMpC3iVpRK5mcDAYAD7VHWbmvklW2NrN3WyL42t9pX3ZgSzj5uPAumVDNncjUVPnUcZ0vEld9OtRSshIYxh8cSxDBIJLV7ycFo/g897YwleHVnc1cr4Z197YA7+M6bXM38Ke5yRJV/E9GyVRwOdCWFiuKQtRKMGUKWIHwQSyS7ThW1R+N0xpJ5nRASSWVTfStrtzWyZlsjr+1qJp5UwkHhxLoqrj5lGvOnVDNjfHnWC+H4JTUMtbIkRIWVtTDGV74mCBG5EPhvIAj8UFW/0evx5cB/Aju8TXeo6g/9jMlvm+tbaYvk/7DTRFJZt72Rp9/Yy6p3DtASiQMwvbac982byPwp1ZxYV5UXk8NSM5grS0KUF9kwVGOGi28JQkSCwPeBdwHbgZdF5DFVfa3Xrg+q6mf8imM4HWiL5n1y2LKvjadf38uf3qxnf1uU8uIgJ08fx4KpNcybPIbqsqKcxhcKCmVFQUrDQUqKgpSFg4RsxJExOeFnC2IpsFFVNwOIyAPAxUDvBDEqJJPK7ubOXIeR0YH2KM++Wc/Tb+xlc30bwYCw+KgazjluAkunj83ZkM9QUCgNBykrcsmgNBy04afG5BE/E8QkYFva/e3ASRn2u0xEzgTeBD6vqtsy7JP3GlojeTU0NRJP8NLb+3n69b2s3nqApMLMCRVcd+YMzpg5ftiXz0wlg9Ii72LJwJi852eCyHSiuPcR9DfA/aoaEZFPAT8Gzj3ohUSuBa4FmDp16lDHedhiiWSPxe1zJanKhl3NPP36Xp7b2EB7NEFtRRGXLZzMOcdN8HVNhHTBgHeaqChIiddCsGRgzMjjZ4LYDkxJuz8Z2Jm+g6ruS7t7N/DNTC+kqiuAFQCLFy/On6/pnt1NnTkdpbSzsYNn3tjLM2/sZU9zhJJwgFOPruXc4ycwZ9IY30ceiUB1WZjKkjClYVtQ3pjRws8E8TIwU0Sm40YpLQM+lL6DiNSp6i7v7vuBDT7G44uOaILG9tjAOw6xls4Yz21s4OnX9/L67hYEmDelmg+fdBSnzBg3LKOPggFhXEUR48qLrCPZmFHItwShqnER+QzwBG6Y6z2q+qqI3AqsVNXHgM+KyPuBOLAfWO5XPH7Z1dQxbO8VSyRZvfUAT7++l5fe3k88qUwdW8Y1p07jrGPH+7pYTrricIDaimKqS8M25NSYUUw0n2dwZbB48WJduXJlrsMAoKkjxlZvlrFfIvEEf9/RxMp3DvCXt+pp7oxTXRrmzGPHc+7xE5hRWz5ss4crSkLUVhRRWTK8HdzGmMMnIqtUdfFgnmMzqQ+RqrK7aeiHtaoqOxo7WLXlAKu3HmD9jmaiiSRFoQBLp43l3OMnsGBK9bCd0hGBMaVhxlcW58WkOWPM8LEEcYgaWqNDVoK7I5pg3Y5GVm05wKotB7pGRE2uKeWi2Uey8KgaZk8cM6ydv6n+hbHlRTYCyZgCZQniEMQTSfa2HHrrQVXZur/dJYStB3htp6t9VBoOMnfyGC5fNJmFU2tyUgzP+heMMSmWIA7B3pbIoMtzt0XirNnWyOqt7tRRQ6srmT1tXBnvnzeRRUfVcEJdVc6+rZcXB6mtLKbK+heMMR5LEIPUGUuw31sPoT+qyuaGNlZ7rYQNu5pJKpQVBZk/pZqrltawcGoNtcM08igT618wxvTHEsQg9TcpLpFUnt/UwMotB3hl6wEOePMjZowv57KFk1l0VA3HHVGZ8zkDgQCMKy9mXIX1Lxhj+mYJYhBaI3FaOuMZH0sklW//4Q2e29hAZXGIBVOrWTjVtRJqynNbITWlKBRwHc9lRda/YIwZkCWIQdjdx6Q4VeX/f3YTz21s4KOnHMWlCyYTzKMDcGlRkPEVxVSV2oprxpjsWYLI0v62KB3RzD3TP39pK79fv5tLF0zig4umZNwnF6pKQ9RWFFOe47WijTEjkx05spBMKnv6WOvht+t28sDL2zj/hAksP3Xa8AaWgQjUlBdRW1Fky3EaYw6LJYgs1Pex1sOf36xnxbObOWn6WD5zzsycnr4JBoRab2JbrjvBjTGjgyWIAUTjSeozrPWwessBvvPkm5w4sYqb/uG4nPU5pCa21ZSFrX/BGDOkLEEMYE/zwcNaX9/dzP/3+w0cNbaM//c9J+bkVI5NbDPG+M0SRD/ao/GD1nrYur+dW3/zGmPLi/jK+2cNawdwamJbbUUxpUXWv2CM8ZcliH7s6lWtdW9zJzf/ej2hoHDr+2dTUzY88xuKwwHGlIapKSuy1dqMMcPGEkQfmtpjtEcS3fc7Ytz82Kt0xhL8x6VzOXKMv4X0UklhTGnYymAYY3LCEkQGqsrutGGt7dE4X/nNq9S3RLj14llMry335X1LvKRQZUnBGJMHLEFkkL7WQyyR5Ou/28Dm+lb+z3tOZNbEMUP6XqVFAaq8loLNWzDG5BNLEL2kr/WQqq+0bnsTnz9/JkumjR2S9ygtCnadPrI+BWNMvrIE0cseb60HVeXOP2/i+U37+Php0zn3+CMO63XLil1SqCqxpGCMGRksQaTpjCU44K31cN+LW3ni1d1cvnAyH1gwadCvJeLWfkj1KVhZbWPMSGMJIk1qrYfH1u7kwZXbuODEI/joKUcN6jUqSkJeSyFkJS+MMSOaJQhPS2eMls44f3pjL3f/ZTOnzBjHp88+JqvyFaGgUFNWRE25dTQbY0YPSxB4w1qbOlm5ZT+3P/UWcyeN4QsXDFxfqbw4yLhyW2fBGDM6WYIADrTHeGVrI//x+9eZNq6Mf3vPCX12JAcDQk25m9VscxWMMaNZwSeIRFJ5cfM+vvrbV6ktL+Ir75tFWdHBv5ay4iDjyosYU2pVU40xhaHgE8TabY3826PrKQ4FufXi2VSn1VcKBKCmzK2xYK0FY0yhKegEsauxgxt+vppoPMk3Lp3DEVWuvlJpUXdrIZBHa0sbY8xw8nUcpohcKCJviMhGEflihseLReRB7/EXRWSan/Gka+mM8dF7XmJfW5Sb33si02rLGVtRxDETKjhmQgU15UWWHIwxBc23FoSIBIHvA+8CtgMvi8hjqvpa2m4fBw6o6jEisgz4JnClXzGldMYSfPx/XmZzQxv/fvEszjthAtVlRTlbFc4YY/KRn6eYlgIbVXUzgIg8AFwMpCeIi4GveLcfAe4QEVHtvYZbt1d3NjPr5scPK7B4UonEk3zzsjlcuWTqYb2WMcaMVn4miEnAtrT724GT+tpHVeMi0gSMAxrSdxKRa4Frvbutr/37RW8MRYDLvgnLem6q7f3eeW6kxQsjL+aRFi9YzMNhpMULcNxgn+Bngsh0vqZ3yyCbfVDVFcCKoQiqPyKyUlUX+/0+Q2WkxQsjL+aRFi9YzMNhpMULLubBPsfPTurtwJS0+5OBnX3tIyIhYAyw38eYjDHGZMnPBPEyMFNEpotIEe5szmO99nkMuNq7fTnwdH/9D8YYY4aPb6eYvD6FzwBPAEHgHlV9VURuBVaq6mPAj4CfishGXMthWd+vOCx8P401xEZavDDyYh5p8YLFPBxGWrxwCDGLfWE3xhiTiS1YYIwxJiNLEMYYYzIqyAQhIveIyF4RWZ+2bayI/FFE3vKua3IZY28iMkVEnhGRDSLyqoh8ztuel3GLSImIvCQia714v+ptn+6VVXnLK7NSNNBrDTcRCYrIKyLyW+9+XscsIu+IyN9FZE1qKGO+fi4ARKRaRB4Rkde9z/MpeR7vcd7vNnVpFpEb8zzmz3v/d+tF5H7v/3HQn+OCTBDA/wAX9tr2ReApVZ0JPOXdzydx4J9V9QTgZOAGETmR/I07ApyrqvOA+cCFInIyrpzKd7x4D+DKreSbzwEb0u6PhJjPUdX5aWPz8/VzAfDfwOOqejwwD/e7ztt4VfUN73c7H1gEtAO/Ik9jFpFJwGeBxao6GzdIKFXKaHCfY1UtyAswDVifdv8NoM67XQe8kesYB4j/17g6V3kfN1AGrMbNpG8AQt72U4Anch1fr1gn4/7ZzwV+i5vMme8xvwPU9tqWl58LoAp4G2+ATL7HmyH+C4C/5nPMdFeoGIsbqfpb4B8O5XNcqC2ITI5Q1V0A3vWEHMfTJ6/q7QLgRfI4bu9UzRpgL/BHYBPQqKpxb5ftuA9zPrkd+Bcg6d0fR/7HrMAfRGSVV5YG8vdzMQOoB+71TuP9UETKyd94e1sG3O/dzsuYVXUH8G1gK7ALaAJWcQifY0sQI4yIVAC/AG5U1eZcx9MfVU2oa5ZPxhVvPCHTbsMbVd9E5L3AXlVdlb45w655E7PnNFVdCFyEO/V4Zq4D6kcIWAjcqaoLgDby5NTMQLxz9u8HHs51LP3x+kIuBqYDE4Fy3GejtwE/x5Yguu0RkToA73pvjuM5iIiEccnhPlX9pbc57+NW1UbgT7i+k2qvrApkLr+SS6cB7xeRd4AHcKeZbie/Y0ZVd3rXe3HnxpeSv5+L7cB2VX3Ru/8ILmHka7zpLgJWq+oe736+xnw+8Laq1qtqDPglcCqH8Dm2BNEtvezH1bhz/HlDRAQ383yDqt6W9lBexi0i40Wk2rtdivvQbgCewZVVgTyKF0BVv6Sqk1V1Gu5UwtOq+mHyOGYRKReRytRt3Dny9eTp50JVdwPbRCRVWfQ83BIAeRlvL1fRfXoJ8jfmrcDJIlLmHTdSv+PBf45z3aGSo06c+3Hn5mK4bzQfx51rfgp4y7sem+s4e8V8Oq5JuA5Y413ena9xA3OBV7x41wM3e9tnAC8BG3FN9eJcx9pH/GcDv833mL3Y1nqXV4F/87bn5efCi20+sNL7bDwK1ORzvF7MZcA+YEzatryNGfgq8Lr3v/dToPhQPsdWasMYY0xGdorJGGNMRpYgjDHGZGQJwhhjTEaWIIwxxmRkCcIYY0xGliDMiCAi49Kqae4WkR1p97Oqrioi96aNv+9rnxtE5MNDFPPFXnxrReQ1EfnEAPuf6xU0zPRYnYj8Lu21HvO2TxGRB4ciXmN6s2GuZsQRka8Arar67V7bBfeZTmZ84jASkWJcUbrFqrrTu3+Uqr7Zz3O+BjSo6u0ZHvsRbhbv9737c1V1nU/hGwNYC8KMcCJyjFfz/i5cxdg6EVkhIiu9evg3p+37nIjMF5GQiDSKyDe8b+QviMgEb5+viciNaft/Q9y6Fm+IyKne9nIR+YX33Pu995rfK7QxuDpO+wFUNZJKDiJyhIj80nveSyJysogcDXwCuMlrdZza6/XqcJM68V5vXdrPv8a7fW9aq6pBRP7N2/5F733Wpf8+jBmIJQgzGpwI/EhVF6irZPlFdesizAPe5a2b0dsY4M/q1qt4AfhYH68tqroUuAlIHVz/CdjtPfcbuMq6Pairi/QEsEVEfi4iV4lI6v/tu8C3vBivAH6oqpuAHwL/qW7tged7veQdwI9F5GkR+XKqBlCv97xGXXHES3ClnX8iIu8GpuJKrc8HTs2QfIzJyBKEGQ02qerLafevEpHVuBbFCbgE0luHqv7eu70Ktz5IJr/MsM/puGJ+qGqqxMVBVHU5bs2OlbiKpSu8h84H7vK++T8K1Hj1qvqkqr8DjsbV4zoReEVExvXez3udh4HrVXUbrjbTRbiyJ6uBY4Bj+3svY1JCA+9iTN5rS90QkZm4FeGWqmqjiPwMKMnwnGja7QR9/y9EMuyTqQR4Rt6poHUi8nNcscJPeM9fqqrpMeC6UPp9rX3AfcB9IvI4LlH1Tk53Aw+o6jNpsX5NVX+UbczGpFgLwow2VUAL0OydhvkHH97jOdypIURkDhlaKCJSJT3XZZgPbPFuPwnckLZvqv+iBajM9IYicl6qlSEiVbha/1t77fM5INyr8/4J4ONepVdEZLKI1Gb5c5oCZy0IM9qsxpU2Xg9sBv7qw3t8D3d+f533futxq3alE+BLInI30AG00t3PcQNwp4hcg/sffMbb9mvgYRG5FLihVz/EEuAOEYnhvtjdqaqviMgxaft8AWhPdVoDd6jqD0XkeOBvXgulBfgQro/CmH7ZMFdjBkncoishVe30Tmn9AZip3cs5GjMqWAvCmMGrAJ7yEoUA11lyMKORtSCMMcZkZJ3UxhhjMrIEYYwxJiNLEMYYYzKyBGGMMSYjSxDGGGMy+r8+1HVLdpLeQQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f457f48c9e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_learning_curve(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For an even more complex model, we still converge, but the convergence only happens for *large* amounts of training data.\n",
    "\n",
    "So we see the following:\n",
    "\n",
    "- you can **cause the lines to converge** by adding more points or by simplifying the model.\n",
    "- you can **bring the convergence error down** only by increasing the complexity of the model.\n",
    "\n",
    "Thus these curves can give you hints about how you might improve a sub-optimal model. If the curves are already close together, you need more model complexity. If the curves are far apart, you might also improve the model by adding more data.\n",
    "\n",
    "To make this more concrete, imagine some telescope data in which the results are not robust enough.  You must think about whether to spend your valuable telescope time observing *more objects* to get a larger training set, or *more attributes of each object* in order to improve the model.  The answer to this question has real consequences, and can be addressed using these metrics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recall, Precision and F1-Score\n",
    "\n",
    "Intuitively, [precision](http://en.wikipedia.org/wiki/Precision_and_recall#Precision) is the ability\n",
    "of the classifier not to label as positive a sample that is negative, and\n",
    "[recall](http://en.wikipedia.org/wiki/Precision_and_recall#Recall) is the\n",
    "ability of the classifier to find all the positive samples.\n",
    "\n",
    "The  [F-measure](http://en.wikipedia.org/wiki/F1_score>)\n",
    "($F_\\beta$ and $F_1$ measures) can be interpreted as a weighted\n",
    "harmonic mean of the precision and recall. A\n",
    "$F_\\beta$ measure reaches its best value at 1 and its worst score at 0.\n",
    "With $\\beta = 1$,  $F_\\beta$ and\n",
    "$F_1$  are equivalent, and the recall and the precision are equally important."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import zipfile\n",
    "with zipfile.ZipFile('../datasets/titanic.csv.zip', 'r') as z:\n",
    "    f = z.open('titanic.csv')\n",
    "    titanic = pd.read_csv(f, sep=',', index_col=0)\n",
    "titanic.head()\n",
    "\n",
    "# fill missing values for Age with the median age\n",
    "titanic.Age.fillna(titanic.Age.median(), inplace=True)\n",
    "\n",
    "# define X and y\n",
    "feature_cols = ['Pclass', 'Parch', 'Age']\n",
    "X = titanic[feature_cols]\n",
    "y = titanic.Survived\n",
    "\n",
    "# train/test split\n",
    "from sklearn.cross_validation import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)\n",
    "\n",
    "# train a logistic regression model\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "logreg = LogisticRegression(C=1e9)\n",
    "logreg.fit(X_train, y_train)\n",
    "\n",
    "# make predictions for testing set\n",
    "y_pred_class = logreg.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[107,  21],\n",
       "       [ 52,  43]])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "confusion_matrix(y_test, y_pred_class)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![f1score](images/Precisionrecall.svg.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "precision_score  0.671875\n",
      "recall_score     0.45263157894736844\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import precision_score, recall_score, f1_score\n",
    "print('precision_score ', precision_score(y_test, y_pred_class))\n",
    "print('recall_score    ', recall_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### F1Score\n",
    "\n",
    "The traditional F-measure or balanced F-score (F1 score) is the harmonic mean of precision and recall:\n",
    "\n",
    "$$F_1 = 2 \\cdot \\frac{\\mathrm{precision} \\cdot \\mathrm{recall}}{\\mathrm{precision} + \\mathrm{recall}}.$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "f1_score     0.5408805031446541\n"
     ]
    }
   ],
   "source": [
    "print('f1_score    ', f1_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "We've gone over several useful tools for model validation\n",
    "\n",
    "- The **Training Score** shows how well a model fits the data it was trained on. This is not a good indication of model effectiveness\n",
    "- The **Validation Score** shows how well a model fits hold-out data. The most effective method is some form of cross-validation, where multiple hold-out sets are used.\n",
    "- **Validation Curves** are a plot of validation score and training score as a function of **model complexity**:\n",
    "  + when the two curves are close, it indicates *underfitting*\n",
    "  + when the two curves are separated, it indicates *overfitting*\n",
    "  + the \"sweet spot\" is in the middle\n",
    "- **Learning Curves** are a plot of the validation score and training score as a function of **Number of training samples**\n",
    "  + when the curves are close, it indicates *underfitting*, and adding more data will not generally improve the estimator.\n",
    "  + when the curves are far apart, it indicates *overfitting*, and adding more data may increase the effectiveness of the model.\n",
    "  \n",
    "These tools are powerful means of evaluating your model on your data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  },
  "name": "_merged"
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
